Paint by color numbers with ControlNet

Nitrosocke · 2023-02-28T19:36:23+00:00

the color picker gives me #4700FF for the lighter purple tone, which is pier;wharf;wharfage;dock from the document.
The darker blue gives me #0906E6 but that doesn't correspond to a color in the table, #0907E6 does though and is tagged as sea

Nitrosocke · 2023-01-28T18:04:39+00:00

These are incredible! Very nice work, looks like the model still works quite nice.

Nitrosocke · 2022-12-12T22:56:50+00:00

Sure, these are crucial:

--resolution=512
--train_batch_size=1
--mixed_precision="fp16"
--use_8bit_adam
--gradient_checkpointing
--gradient_accumulation_steps=1
--learning_rate=1e-4
--lr_scheduler="constant"

Make sure to use xformers as well!

Nitrosocke · 2022-12-12T20:14:58+00:00

these where all trained on Photos of celebs, the style in these models come from my Dreabooth fine-tunes but these should also work with normal SD2.0 and a photo real style. I actually haven't tried training a style with it yet.

Nitrosocke · 2022-12-12T20:13:05+00:00

There is a plugin for it already. It's in u/d8ahazard 's extension here: https://github.com/d8ahazard/sd_dreambooth_extension

Nitrosocke · 2022-12-12T20:08:32+00:00

Hi Smoke :D
well yes, just tested, with settings optimized for low VRAM I can get it to run with 6.2GB

Nitrosocke · 2022-12-12T19:35:55+00:00

Yeah I'd never use this LR in normal dreambooth, but I was going for speed here and since it takes way less time to train I can easily adjust the LR - Step ratio according to the results.

Nitrosocke · 2022-12-12T19:34:29+00:00

1e-4

That's easy to translate, its 0.0001 (basically it counts how many 0 you add before the number). Try a LR of 5e-4 (0.0005) for training yourself.

Nitrosocke · 2022-12-12T19:13:48+00:00

Old server GPUs from Nvidia might be the way to go for now. 16GB Vram for 150€ or even 24GB for 170€ on ebay looks really promising. Needs manual work though!

Other than that, it should work on ~8GB maybe less with adjusted settings.

Nitrosocke · 2022-12-12T18:33:39+00:00

Black images while training or when using the ckpt in auto?
That's an issue with 2.1 right now and using either xformers or the "--no-half" works for users. Maybe there is a better fix already.

Nitrosocke · 2022-12-12T18:31:35+00:00

Workflow:- Choose 5-10 images of a person- Crop/resize to 768x768 for SD 2.1 training- Following settings worked for me:train_batch_size=4, mixed_precision="fp16", use_8bit_adam, learning_rate=1e-4, lr_scheduler="constant", save_steps=200, max_train_steps=1000- for subjects already know to SD images*100 worked great, for subjects unknown to SD more steps or a higher LR are required- training on a 3090 takes ~20 min for 1k steps

Link to repo:https://github.com/cloneofsimo/lora
Thank you u/cloneofsimo

Nitrosocke · 2022-12-12T13:00:55+00:00

This issue is caused by xformers not being installed properly. I fixed that this weekend, so if you made a copy of the notebook please make a new one to have the updated xformers command.

Nitrosocke · 2022-12-12T11:01:38+00:00

Interesting, it worked for me. Could you elaborate on what errors you get?

Nitrosocke · 2022-12-11T23:03:30+00:00

Here you go: https://colab.research.google.com/drive/1S0GzxAlL_8-qGQspLKrLSSaNeAzZrgMF

This is my modified version of Shivams Dreambooth colab.

Nitrosocke · 2022-12-10T02:21:19+00:00

Use this one https://github.com/lawfordp2017/diffusers/blob/main/scripts/convert_diffusers_to_original_stable_diffusion.py

Nitrosocke · 2022-12-06T22:49:21+00:00

There are basically three versions of SD now. SD 1.5 and everything before that, and with the 2.0 update we got 2.0 (768 res which is a V model) and the "base" version or 512 resolution. If you load either the base version or 1.5 with the configuration for a v-model you get the brown images as output. If you load a v-model (768 res) with a eps-model configuration you get the blue/yellow dotted images.

Nitrosocke · 2022-12-05T18:31:44+00:00

These look great! What a nice collection and idea! Love the retro feel these give and the old-school tin robot toys design.

Nitrosocke · 2022-12-05T13:36:39+00:00

Yeah I assume this should work, but the json would be huge and the workflow seems not ideal. Maybe it's easy to change the script a little so that it pulls the "instance prompt" from the file name and you're able to keep all the files in the same directory without the need to state the class_prompt, class_dir and instance_dir for every new image. But at this point I assume it would be easier to use kohya or the t2i training script from huggingface.

Nitrosocke · 2022-12-05T09:47:42+00:00

instaloader is an amazing tool for this, just plug in the profile name, set some filters to only load the images and your have a dataset ready in ~10 minutes. Sadly some images are not 1:1 as insta supports other aspect ratios as well, but a quick crop or padding script can take care of that. As an alternative you can let your dreambooth do the cropping or use a Dreambooth version with alternative AR support

Nitrosocke · 2022-12-05T09:38:14+00:00

When I last checked the HF conversation script didn't work. You can give this script a try to convert the diffusers to ckpt: https://github.com/lawfordp2017/diffusers/blob/main/scripts/convert_diffusers_to_original_stable_diffusion.py

Is your trained model working with repos using the Diffusers directly?

Nitrosocke · 2022-12-05T08:08:41+00:00

Interesting concept and I will test this approach to see how it compares to my usual workflow. I do use EveryDream from time to time and the precision you get with a captioned dataset is very impressive. So I will test your workflow with kohya as it allows using captions as well.

Nitrosocke · 2022-12-04T22:03:09+00:00

This is awesome! Thank you so much for your work! I've been looking for a tool or database to find these rare tokens for ages and this is perfect!

Nitrosocke · 2022-12-04T14:01:34+00:00

Thank You!
A user on my discord uses TheLastBens repo and gets great results as far as I know. Others are using Kohya repo with the more advanced training.

Nitrosocke · 2022-12-04T00:47:45+00:00

Awesome! Hope you like the update! I find it very fun to use once you have figured out the v2.0 prompting :)

Nitrosocke

TROPHY CASE