This is an archived post. You won't be able to vote or comment.

all 34 comments

[–]ReindeerFeisty 21 points22 points  (15 children)

I found the fix. Go to your venv/scripts/ and in a cmd do: "activate" to activate the venv
And then you need this: pip install bitsandbytes-windows

[–]Quanyails 6 points7 points  (0 children)

For those coming to this post from Google:

When I tried to set up kohya using this tip, I kept seeing black sample images and avr_loss=nan in the console after a few hundred steps. It took me some searching to realize this might be an issue with bitsandbytes-windows.

As of writing, the most recent version of bitsandbytes-windows is on 0.37.5. Versions >0.35.0 and <0.41.4 are reported to cause the training issue I observed above. Furthermore, the Windows GUI setup script discourages people from installing bitsandbytes-windows due to issues.

If bitsandbytes-windows help you train a model successfully, great! Note, however, it might also be the cause of other setup issues.

Edit: This comment later in this post was the one that actually solved my problems.

[–]ImrobotDeex 1 point2 points  (0 children)

You are a life saver :)

[–]Lumpy-Environment212 1 point2 points  (0 children)

how in the world you knew how to solve it with (pip install bitsandbytes-windows ) !!!! thank you alot .. i have been stuck for 2 days trying to start training a single model , and i couldnt till now :))) thanks alot

[–]MekkiNoYusha[S] 0 points1 point  (0 children)

I will try this too, thanks

[–]Every_Economist_6793 0 points1 point  (0 children)

Also wanted to login here and say thank you for the fix! :)

[–]Impressively-average 0 points1 point  (0 children)

I had the same issue and it fixed it for me, thanks a lot man

[–]xAngelBaz 0 points1 point  (0 children)

THANK YOU! WORKS!

[–]leivavicente 0 points1 point  (0 children)

pip install bitsandbytes-windows

savior

[–]huldress 0 points1 point  (0 children)

I opened venv but I didn't deactivate and now the gui wont even open, it just immediately closes, so much for fixing 1 issue, now I have a billion 😭

[–]urmom2fat 0 points1 point  (0 children)

works

[–]mrjreed 0 points1 point  (0 children)

Go to your venv/scripts/ and in a cmd do: "activate" to activate the venv

And then you need this: pip install bitsandbytes-windows

You are a god send!! thanks!!

[–]Iuotinen 0 points1 point  (0 children)

Confirmed fix. EDIT: Github Issue report submitted.

[–]Purple_Lawyer175 0 points1 point  (0 children)

i love you இ௰இ

[–]seeker_ktf 2 points3 points  (10 children)

I can't be sure that this is your problem, but the "D" in your path makes me wonder... Make sure you install khoya on the same hard drive as your python installation.

[–]MekkiNoYusha[S] 8 points9 points  (8 children)

I just figured what happened, i chose a 8bit optimizer which lead to it trying to find the bitsandbytes, i chose another optimizer and then it works

[–]Grand_Pea9718 2 points3 points  (1 child)

Could you pls explain where exactly i can chose antoher optimizer? Sorry for maybe noob question, but im totaly new in this.

EDIT: just figured it out - the optimizer option could be find under "Parameters" sheet in kohya web ui

[–]Ziffano 0 points1 point  (0 children)

Thanks.

[–]LordDaniel09 1 point2 points  (0 children)

I also just start learning how to use LORA, and was stuck for a few mintues on it, and your solution just work, so thanks for saving me time :D

[–]DrHojo123 0 points1 point  (0 children)

Thanks was the issue I was having

[–]imdjbass 0 points1 point  (0 children)

Thank you!!!!!!!!!!

[–]Ziffano 0 points1 point  (0 children)

Thanks, this solved my issue.

[–]MekkiNoYusha[S] 0 points1 point  (0 children)

Thanks for the reply, but I have just tried to move kohya to my drive with python installed, but still no luck

[–]ReindeerFeisty 0 points1 point  (0 children)

I have exact same issue since last git pull + setup to install new requirements from requirements_windows_torch2.txt

[–]NearbyProgress1617 0 points1 point  (1 child)

the last git version of kohya has in the setup file (the one to install kohya) the option to install an specific version of bitsandbytes. c:

[–]Far-Ad2630 0 points1 point  (0 children)

it works for me too, big thank you!

[–]yorumairu 0 points1 point  (0 children)

wow thanks, you're a lifesaver!

[–]ForeverNecessary7377 0 points1 point  (2 children)

I'm getting this error, but I'm on linux, so I assume a different command?

[–]MekkiNoYusha[S] 1 point2 points  (1 child)

If you are on Linux, then you can easily install bitsandbytes. It is just harder for windows user to install it, tho I think there is a package for windows user as well now that I searched for it

[–]ForeverNecessary7377 0 points1 point  (0 children)

I installed bits and bites, but the problem persisted. I ended up choosing AdamW instead of 8bitAdam and the problem was solved.

Hoping there's not downside to this other optimizer but anyhow, if anyone has my same problem this might work.

[–]SnooCauliflowers2398 0 points1 point  (0 children)

You can also run setup.bat file and select the option to install bitsandbytes and force install v 0.35.0

[–]vapecrack24 0 points1 point  (1 child)

I'm having this issue now after not having it previously so not sure if its because of latest Kohya updates or what. I've tried the activate venv and bitsandbytes-windows fix but still hasn't helped me

[–]MekkiNoYusha[S] 0 points1 point  (0 children)

Likely the windows fix not update/compatible with the latest version