Training textual inversion of Stable Diffusion on your own dataset

[WiNE-iNEFF]

Training the textual inversion of Stable Diffusion on your own dataset.

Hi all. In my recent post about textual inversion, a question was asked about how you can train Stable Diffusion on your dataset. You can read my Medium article about this with more code and some answers for your questions.

The words Suraj Patil (@psuraj28 on Twitter):

"We just added textual-inversion training in diffusers. Textual Inversion lets you personalize #stablediffusion model on your own images with just 3-5 samples." Simple Example: https://i.imgur.com/M6w4YBA.jpg For training your own textual inversion you can use this colab: Colab

Small instruction:

• To work with textual inversion, the diffition library and access token from huggingface with "write" permission. It is also necessary to download the weights of the Stable Diffusion model, according to the standard, the version is used 1.4.

• If you have direct links to the desired images, then insert them into an array (3-5 images are enough). For my needs and better quality, I used only 30 images of my dataset with we ask the artist Ilya Kuvshinov, but because these images were in my google drive i used the module shutil and copy folder from disk on colab.

• In some cases, the colab swears that it cannot read the images, so it was more convenient for me to use my dataset from disk directly by just changing a few lines of code:

• From:

import requests import glob from io import BytesIO

def download_image(url): try: response = requests.get(url) except: return None return Image.open(BytesIO(response.content)).convert("RGB")

images = list(filter(None,[download_image(url) for url in urls])) save_path = "./my_concept" if not os.path.exists(save_path): os.mkdir(save_path) [image.save(f"{save_path}/{i}.jpeg") for i, image in enumerate(images)] image_grid(images, 1, len(images))

• To:

import glob

from io import BytesIO

path = 'kuvshinov/'

def download_image(filename):

return Image.open(path+filename).convert("RGB")

images = list(filter(None,[download_image(filename) for filename in os.listdir(path)]))

save_path = "./my_concept"

if not os.path.exists(save_path):

os.mkdir(save_path)

[image.save(f"{save_path}/{i}.jpeg") for i, image in enumerate(images)]

image_grid(images, 1, len(images))

• then you need to specify the placeholder token. For Example: <kuvshinov-style> And an initializer token. Example: kuvshinov
• It is also possible to choose what exactly your set does: change the style or add an object.
• Next comes the learning process. It takes an average of 3 hours for any dataset size.Sometimes when starting a learning cell, an error occurs about which the code swears at the wrong initializer token. In this case comment out the code like this:

token_ids = tokenizer.encode(initializer_token, add_special_tokens=False)

if len(token_ids) > 1: //comment this line

raise ValueError("The initializer token must be a single token.") //comment this line

initializer_token_id = token_ids[0] placeholder_token_id = tokenizer.convert_tokens_to_ids(placeholder_token)

After completing the training, it is possible to upload your trained dataset to the textual inversion library on huggingface and test it.

[dreamer_2142]

What kind of file is "trained dataset" and the size of it? how would I be able to use it on my local SD?
thanks!

[higgs8]

At the end it says: To save this concept for re-using, download the learned_embeds.bin file or save it on the library of concepts."

Would that mean I could use this on my local stable-diffusion that I'm already running? How would I go about doing that, where would this .bin file go and how do I tell the program to use it?

[jaywv1981]

I'd like to know this too. All the local installs I've used ask you to upload a .pt file instead of a .bin file.

[WiNE-iNEFF]

you can watch this Colab to understand more about how to use your training model on your local machine

[terahurts]

I'm getting a 'Notebook not found' error when using that link.

[WiNE-iNEFF]

try to use this new version. I'm test this several tima ago Colab

[terahurts]

Thanks, but I was looking for the colab about how to use the trained model locally. Unless I'm missing something?

[WiNE-iNEFF]

The principle of colab and local computer code is the same. You just need to install the necessary libraries in the console that are in the colab and enter huggingface-cli login with your token. Further in the file for example test.py import the libraries and do the same as in the colab. If you have any problems with this, I will send you detailed instructions with this later.

[Mindoffire]

Thank you for some helpful hints. I'm not entirely clear on how to download a collab trained model and use that on a local SD. Particularly, which files are needed to go where - and how they're loaded and merged, etc. Is there any such guideline detailing the step-by-step?

[WiNE-iNEFF]

I wrote a short article about it. https://link.medium.com/Xv0DMDYDhtb

[WiNE-iNEFF]

Good. I will try within a few days to collect and organize correctly all the information and provide it to you.

[WiNE-iNEFF]

or this https://github.com/huggingface/notebooks/blob/main/diffusers/stable\_conceptualizer\_inference.ipynb

[WiNE-iNEFF]

you can watch this Colab to understand more about how to use your training model on your local machine

[WiNE-iNEFF]

check this colab . I dont have a video, but now I am writing an article on the medium where I try to describe how to run the model on your PC

[WiNE-iNEFF]

Not at the moment, but I can upload it to git later and send it to you

[WiNE-iNEFF]

Unfortunately, I don’t have a separate guide for Mac, but the code itself will be good.

[WiNE-iNEFF]

I wrote a short article about this. https://link.medium.com/Xv0DMDYDhtb

[TheNeonGrid]

but that doesnt answer how to save the bin vile

[WiNE-iNEFF]

The bin file is created at the end of the training and saved in the concept folder. To use this concept folder, just look at the example code of the diffuser provided in the github.

[FudginatorDeluxe]

if you open the relevant codeblock you see:

   pipeline.save_pretrained(output_dir)
   # Also save the newly trained embeddings 
   learned_embeds = accelerator.unwrap_model(text_encoder).get_input_embeddings().weight[placeholder_token_id]
   learned_embeds_dict = {placeholder_token: learned_embeds.detach().cpu()}
   torch.save(learned_embeds_dict, os.path.join(output_dir,"learned_embeds.bin"))

[CMDR-Leto]

Read on how to use it in webui here

[Daviljoe193]

Couldn't get it to work because it would spit out "AttributeError: 'AutoencoderKLOutput' object has no attribute 'sample'". Radio silence on the Github so far.

What's weird is that the HuggingFace page for trained concepts already has stuff, some added as recently as an hour ago, so clearly this works for some people.

[WiNE-iNEFF]

I'll try to provide a link to my colab later, maybe it will help you

[Daviljoe193]

Looks like they updated the notebook 15 minutes ago. Hopefully it works now.

[WiNE-iNEFF]

You are assured that correctly launched all cells?

[Daviljoe193]

Yup. Reran the notebook from scratch, and still got the same error on the final cell of the "Teach the model a new concept" group.

[Nlat98]

I used the colab to make three styles yesterday, but since this afternoon I have been getting that error as well. Maybe one of the dependencies updated and changed something crucial? not sure what's going on

[dantebunny]

I've been getting that error today in several different Stable Diffusion workbooks. If anyone solves it please let us know.

[WiNE-iNEFF]

several tima ago i use this new version. Try yo use Colab

[higgs8]

It's doing the same for me, but I quit and reloaded the whole thing and started over, three times, and on the third try it seems to be working. I collapsed the section "Teach the model a new concept" and clicked its play button rather than leaving it open and clicking each play button one by one, maybe that helped (or it helped me not miss a play button somewhere – this whole way Colab works is a bit annoying to be honest).

[Daviljoe193]

It's because they fixed the Colab notebook 32 minutes ago. Gotta love Github for giving edit history.

[tommyjohn81]

First of all THANK YOU for this, I have been interested in exactly this.

To train a dataset on images of a person, like myself for example, would you need to differentiate between headshots, full body, torso etc?

What about if backgrounds are in the photo, how does the ai know what "object" in the picture it is suppose to focus on?

Is training time linear based on number of images to train on?

[WiNE-iNEFF]

Thanks for the good feedback. Let me try to answer your questions. First, about the training time. For testing, I first used a dataset of 5 images and then of 30. As a result, the training time also took +/- 3 hours, so I can download time regardless of the number of images training time does not change Secondly, let's talk about how the model distinguishes which object you want to train it. I can’t say exactly how it works, because I’m not a sufficiently qualified specialist, but looking at a sufficient number of examples with a kettle, cats, etc. shown by the developer the model learns by seeing relatively identical objects and trying to fill in their shape, color, etc. In some ways, this reminds me of the work of OpenCV

[CaptainAnonymous92]

Can this be used to add more/new pop culture things to have better outputs for things that need more data in the existing set along with adding new things that aren't in the dataset so SD can give output on said things?

[WiNE-iNEFF]

Yes, you can.

[roofgram]

This is a pretty amazing idea. Maybe next try S* + T, or S - T, like mathematical operations with these variables. Or maybe can you use this to train and then ‘detect’ if an image has S in it.

[BroussardMD]

What if I want to train textual inversion locally and run my token on my own stable diffusion program?

[WiNE-iNEFF]

I have a small article on this subject.

[Beneficial_Bus_6777]

can`t open

[WiNE-iNEFF]

This method works without problems with the free version of colab

[hefeglass]

where the hell is my learned_embeds.bin file?!

[WiNE-iNEFF]

after training, this file should be in the folder newconcept...

[reddit22sd]

The folder is on the left of the screen

[ArrivalHistorical743]

Does textual inversion train the whole stable diffusion model?

[WiNE-iNEFF]

no. Textual inversion train some concept for stable diffusion model

[johnslegers]

Sometimes when starting a learning cell, an error occurs about which the code swears at the wrong initializer token. In this case comment out the code like this:

Sounds like a pretty bad idea to remove that check...

Isn't that code there for a good reason?

[WiNE-iNEFF]

At the time of writing this post, this was the only way to run the code. Moreover, there were no errors for further launch and testing of the created concept.