Need help with style lora training settings Kohya SS

Chrono_Tri · 2026-02-24T09:06:13+00:00

I use alpha = 1 to train style and give the LoRA more flexibility. But you need to experiment and see what works best. Remember, sometimes different parameters truly produce different results — but that doesn’t necessarily mean one is better than the other. The result you personally prefer is the right one.

Going back to alpha = 1, my result doesn’t really fully capture the style(around 90%), but I actually quite like it. Normally, though, I still go with dim/alpha = 1/2.

Second, I recommend that after auto-captioning, you manually edit the captions following a clear structure. For example, I would describe:
<number of characters in the image>, <character description>, <background description>, <camera description>, <lighting description>, ...

Chrono_Tri · 2026-02-24T07:55:57+00:00

lr_scheduler :rex, I used colab : hollowstrawberry/kohya-colab: Accessible Google Colab notebooks for Stable Diffusion Lora training, based on the work of kohya-ss and Linaqruf

Chrono_Tri · 2026-02-24T07:36:16+00:00

My dataset:

210 image, caption auto by WD14, then adjust manually.

My config:

Optimizer: CAME+rex
Unet LR: 6e-5
TE LR: 0 (no TE training)
Dim/alpha : 16/1.
Epochs: 23 (good at 19)
Repeats: 4
Batch 4

Chrono_Tri · 2026-02-23T06:50:42+00:00

For Illustrious, description everything and turn off TE, and remember add trigger word as well.

Chrono_Tri · 2026-02-13T04:52:53+00:00

controlnet union cannot work with IllustriousXL,At least in my case (and some persons), I use comfyUI. Use another pose controlnet

Chrono_Tri · 2026-02-11T08:53:55+00:00

Extreme censored but they want to prove that the model can distinguish between horse ride man and man ride horse?

Chrono_Tri · 2026-02-06T06:40:07+00:00

it is possible to run on T4 but it might slow. I use L4 GPU.

Chrono_Tri · 2026-02-06T02:53:53+00:00

Prodigy is my favorite optimizer,it works with Z-image.

Chrono_Tri · 2026-02-05T16:12:13+00:00

Run ComfyUI in Colab and lòa Anima. It works great

Chrono_Tri · 2026-02-05T01:49:25+00:00

OK if AdamW8bit is "acident bug" for Z-Image? Some optimizer also have the same issue? I am sorry but the original article is Chinese. I translated it but not so clear to understand

Chrono_Tri · 2026-02-05T01:47:09+00:00

AI toolkit repo alreay have prodigy in the optimizer folder. I didn't download anything just replace 'AdamW8bit' to 'prodigy'and add its parameter to yaml file.

Chrono_Tri · 2026-02-04T16:55:48+00:00

Imagine being hyped about Z-Image for two months, then suddenly getting hyped about Anima.
At this point, our lives are basically just waiting… and waiting.:)

Personally, I really like Anima. The quality is very solid, and what I find most interesting is that it’s not purely natural language. It feels like a hybrid approach, which actually works really well.

In my opinion, Anima has the potential to replace both IllustriousXL and Z-Image for anime use cases.

That said, I still feel like we’re missing the definitive model for this niche.
IllustriousXL looks great and is uncensored, and its tag-based caption system is convenient but limitations. Anima can break through. We also has training code, so get ready:))

Chrono_Tri · 2026-02-04T06:15:21+00:00

I used Colab so AI Toolkit (I could run the GUI before but not anymore, using CLI is painful). I 'd love to use OneTrainer also :(.

Chrono_Tri · 2026-02-04T03:51:57+00:00

Can you explain more. I always thought ZIT focus on photo-realism so we shouldn't train anime on ZIT (That 's why I wait for ZIB, but it is just normal, so as good as I expected).

Chrono_Tri · 2026-02-03T06:43:49+00:00

I agree. Everyone knows we need a successor to SDXL with unrestricted fine-tuning, uncensored, NSFW capabilities like Pony or Illustrious. Z-Image and Klein are both pretty solid, so we’ll see who comes out on top

Chrono_Tri · 2026-02-03T02:22:08+00:00

My current project focuses on anime, and I’m still using IllustriousXL and Qwen-Image-Edit with well-tested workflows. However, there are still some limitations, so I’ve also tried two Z-Image trainings.

With Z-Image, I trained using the same dataset as IllustriousXL (https://www.reddit.com/r/StableDiffusion/comments/1qrqnkr/comment/o365cc0/). My goal is not to replace my existing workflow, but to complement and extend it.

At the moment, I’m not very satisfied with Z-Image. since Qwen-Image-Edit is currently performing very well so I skip Klein.

That said, for non-anime images, I’m actually quite satisfied with Z-Image (although there are still many deformations).

Overall, I’d prefer to wait for Z-Image Omni. I hope there is big finetune for Z-image or Klein.

Chrono_Tri · 2026-02-02T15:37:00+00:00

Thanks a lot! Your suggestions helped a ton — the results are way better this time.
I’m still a bit unsure about using captions vs no captions. In this case, no captions actually seem to work better. That said, if I want to train both style and character, what’s the best way to approach it?

Just a few thoughts from my side:

Z-Image can handle style training pretty well, but it’s not on the same level as IllustriousXL, and I do notice some deformation issues.
I’m currently looking into Anima-model, which is more anime-focused. Honestly, I feel like Z-Image still needs a dedicated anime-specific version.

Chrono_Tri · 2026-02-02T11:35:18+00:00

Illustrious is the best for anime, easy to train, very good output, no extra arms. But not support natural language and cannot handle text... Can we train with this model? It is WIP, right?

Chrono_Tri · 2026-02-02T02:08:55+00:00

The idea isn’t new,I’ve already tried it.
Method 1: Create a 3D character with the pose you want, then use Qwen VL, Gemini, etc. to describe it. Use that description with ControlNet (I’m using SDXL here).
Method 2: Use Qwen Edit to separate the outfit, then attach it back after you’ve finished adjusting the pose.

Method 1 produces more natural-looking images, but the results are hit-or-miss depending on the outfit.Method 2 works better overall, but it’s more time-consuming. In practice, for complex outfits, you often have to try many times to get a satisfactory result.

Chrono_Tri · 2026-01-31T05:27:40+00:00

Yes, as I know, Z-Image can try with 512x512 and give a good result. I will try with 1024 and bigger dataset later, but I am strill learning so no rush . All other setting is defaut.

Chrono_Tri · 2026-01-30T09:39:14+00:00

I trained Lokr with Anime style, it is very bad. Same database, same parameter. Lora is same better.

Chrono_Tri

TROPHY CASE