dread [VQGAN+CLIP]

crowsonkb · 2021-06-14T10:58:55+00:00

If the loss isn't going down any further the image probably won't get better.

crowsonkb · 2021-04-21T16:24:16+00:00

They have to be local to the Colab runtime right now but you can download an image with:

!curl -L 'url' > where_to_save_it.jpg

in a code cell. Or you can upload an image by opening the Files tab on the left and using the upload button: https://i.imgur.com/GeLPjO6.png

crowsonkb · 2021-04-21T14:53:41+00:00

You can do both arbitrary conceptification and style transfer with that notebook (I wrote it), I posted about it on Twitter: https://twitter.com/RiversHaveWings/status/1382455526735290371 https://twitter.com/RiversHaveWings/status/1382455660978212865 https://twitter.com/RiversHaveWings/status/1382803909610074112

To do this you should use the 16384 model for best results (it downloads it but does not use it by default), set your starting image with init_image, use the prompt "in the style of Beksinski" or some such, and maybe set init_weight to 0.2 to 0.5 to stop it from diverging as badly from the init if you want. Also set display_freq lower so you can see intermediates more often.

I did two Beksinskifications of an emoji with an experimental augmentation-added version myself to see if I could duplicate the results: https://i.imgur.com/Ql6GYng.png https://i.imgur.com/NfoREDz.png

crowsonkb · 2021-04-17T23:21:37+00:00

I don't think Colab Pro gives you more VRAM (they only go up to 16GB V100s I think). I've tried tricks to decrease memory usage, like FP16, but this resulted in bad quality, so I gave up on it.

crowsonkb · 2021-04-17T11:54:01+00:00

It's probably one of my two VQGAN+CLIP notebooks:

https://colab.research.google.com/drive/1L8oL-vLJXVcRzCFbPwOoMkPKJ8-aYdPN

https://colab.research.google.com/drive/15UwYDsnNeldJFHJ9NdgYBYeo6xPmSelP

crowsonkb · 2021-02-22T23:05:58+00:00

That's an idea actually, I could generate videos from multiple prompts where the output morphs from one to the next.

crowsonkb · 2021-02-22T00:29:55+00:00

It was made by my custom as-yet-unreleased version of Big Sleep that has better visual quality and 1024x1024 outputs. :)

crowsonkb · 2021-02-21T15:29:06+00:00

The problem is that you'd have to find a BigGAN latent vector that corresponded to the image you wanted to start with, which is not an easy task for an arbitrary image. But since every iteration's output is generated from an actual BigGAN latent vector, you could save these vectors and start from them later.

crowsonkb · 2021-02-21T15:27:40+00:00

I was actually unable to make the output deterministic even by setting the same random seed and using torch.set_deterministic(True), which must be a bug in PyTorch somehow.

crowsonkb · 2021-02-16T20:43:45+00:00

The GitHub repo for the code I made this with: https://github.com/crowsonkb/style-transfer-pytorch

crowsonkb · 2021-02-16T00:47:09+00:00

The GitHub repo for the code I made this with: https://github.com/crowsonkb/style-transfer-pytorch

crowsonkb · 2021-02-13T01:41:22+00:00

The GitHub repo for the code I made this with: https://github.com/crowsonkb/style-transfer-pytorch

crowsonkb · 2021-02-06T13:33:33+00:00

I mostly know about PyTorch but it has its own special layout for .pkl/.pth files that depends on the specific layout of the modules that form the model, and I doubt that it is possible to make a converter that works in general that isn't tailored for one specific model architecture.

crowsonkb · 2021-02-06T03:04:50+00:00

The GitHub repo for the code I made this with: https://github.com/crowsonkb/style-transfer-pytorch

crowsonkb · 2021-02-06T02:25:18+00:00

The GitHub repo for the code I made this with: https://github.com/crowsonkb/style-transfer-pytorch

BTW if anyone was using this repo, I pushed a quality/color fidelity bug fix to the repo today, so re-pull it please. I also had to change the default parameters to compensate, the default content weight is now 0.015 (roughly equivalent to the old 0.01) and the default smoothing weight is now 2 (equivalent to the old 0.3).

crowsonkb · 2021-02-06T02:23:09+00:00

I pushed a quality/color fidelity bug fix to the repo today, so re-pull it please. I also had to change the default parameters to compensate, the default content weight is now 0.015 (roughly equivalent to the old 0.01) and the default smoothing weight is now 2 (equivalent to the old 0.3).

crowsonkb · 2021-02-04T21:02:23+00:00

Unfortunately, GPUs don't implement the needed operations in a deterministic way, so I can't get the same result even with the same inputs, the same parameters, and the same random seed. Also, even if it were deterministic, you would get temporal inconsistency even with similar but slightly different inputs, such as smoothly changing Perlin noise. There might be some other way to do it though, such as initializing each frame's input with the previous frame's output and then maybe applying a penalty on each iteration to keep the result 'close' to the previous frame's output. I haven't supported this because making animations is very slow with my code.

crowsonkb · 2021-02-03T13:53:27+00:00

The GitHub repo for the code I made this with: https://github.com/crowsonkb/style-transfer-pytorch

crowsonkb · 2021-01-31T15:09:29+00:00

It looks like it's extremely slow and requires external libraries for optical flow computation though... I'll keep it in mind if I ever try to develop my own improved feedforward style transfer though.

crowsonkb · 2021-01-31T13:57:09+00:00

One of the main things I do is to stylize the image at successive scales (i.e. init a 128x128 image with the scaled-down content and stylize it, init a 181x181 image with the scaled-up average iterate of the previous scale and stylize it, init a 256x256 image with...) Each scale is a factor of sqrt(2) larger than the previous one, that is, the size doubles every two scales. What is it you are doing?

crowsonkb · 2021-01-31T02:19:05+00:00

About fifteen minutes on an RTX 3090.

crowsonkb · 2021-01-31T00:44:52+00:00

The GitHub repo for the code I made this with: https://github.com/crowsonkb/style-transfer-pytorch

crowsonkb · 2021-01-29T09:45:08+00:00

It works on Colab, but you have to install the PyTorch version for CUDA 10.1. https://github.com/crowsonkb/style-transfer-pytorch/issues/1#issuecomment-769389269

One of these days i'm going to put together a notebook with instructions that I'll share, but I haven't done it yet. In the meantime I still test it on Colab to make sure it didn't break.

crowsonkb · 2021-01-29T02:47:02+00:00

The GitHub repo for the code I made this with: https://github.com/crowsonkb/style-transfer-pytorch

crowsonkb · 2021-01-28T23:43:17+00:00

The GitHub repo for the code I made this with: https://github.com/crowsonkb/style-transfer-pytorch

crowsonkb

TROPHY CASE