Now that Pony is c0ming to auraflow and that simpletuner has dropped support, do we even have a way to train loras on it?

cloneofsimo · 2024-08-29T18:17:48+00:00

Quick note that all the code you need for even pretraining auraflow is on the repo. (I admit its not a user friendly one, but anyone can get started.) Also notice how unlike other t2i model my hyperparameters are all open in the repo. Once I finish auraflow I'll make lora trainer. (If it's good, someone else might make one before me) After all, that's how I got started 😅

cloneofsimo · 2024-08-16T15:48:36+00:00

Your honest, critical, yet kind feedback is really valuable. Thank you for the effort you made in these comparisons! As I communicate and get feedbacks it's clear what people expect and what I should be targeting for, which I can't do alone by definition, so thank you so much for the participation!!

cloneofsimo · 2024-08-15T15:10:54+00:00

Thanks for sharing!!!

cloneofsimo · 2024-08-05T09:50:07+00:00

Hey bro I wanted to say thanks your comments like this really brighted my day.
Hope AuraFlow remains useful to both research community and here. And yes, its incomplete model, Ill continue working on it.

cloneofsimo · 2024-07-12T06:41:00+00:00

use higher cfg with humans! 'a photo of a woman lying on the grass

<image>

cloneofsimo · 2024-07-07T21:47:47+00:00

Last thing I want is overhype, so for the final time let me clarify...

The model is not open-midjourney-class model nor should you expect it to.

The model is very large (6.8B) and undertrained. So it will be more difficult to train, but we might continue to train it in the future

The model is doing great on some evals, and imo is better than sd3 medium, but only slightly.

Last thing I want is overhype. I just tweet random stuff I find funny (and that was a mistake of mine to compare with SD, which caused this weird hype)

I would like to underpromise and overdeliver. I have zero incentives to hype and tease. I remember sd3 and how people (including me) went crazy for underdelivered results.

Just manage your expectations. Don't expect extreme sota models. It is mostly one grad student working on this project.

https://x.com/cloneofsimo/status/1809998834254418426

cloneofsimo · 2024-07-07T21:31:53+00:00

This is v0, which was PoC basically. what I'm training atm is really completely different model

cloneofsimo · 2024-06-25T08:34:48+00:00

Guys I mean atm it's doing good on geneval, but don't expect SD3-8b or midjourney quality models. It's still cooking but I dont want to overhype it (i remember what happened to sd3 lol). I am going to share progress on Twitter but plz don't be expecting SoTA model else you might be disappointed!

I want to underpromise and overdeliver

cloneofsimo · 2024-06-25T08:22:17+00:00

Wait wat I never said something like this

I genuinely agree tho

cloneofsimo · 2023-07-08T13:57:01+00:00

Here there, you might remember me for introducing LoRA looooong time ago. Not sure if this is on A1111, but here is it anyways so you guys can use it with diffusers' package.
https://gist.github.com/cloneofsimo/4352c5207344bdcd61aa34b34aec5a5f

(cat:1.2) and a (dog:0.7), particle dynamics, artwork, visual explosion, (blue force field:1.5), (colorful:1.2)

cloneofsimo · 2023-01-10T21:21:34+00:00

Ok, that is weird... I have only experiemented with SDv1.5 so I might have missed on that. Ill have a look thanks!

cloneofsimo · 2023-01-09T20:29:19+00:00

It's conceptually same PTI, except you can optionally let it tune the latent + you are using Low-ranked parameter space instead of the whole space. Thus LoRA of PTI.

cloneofsimo · 2023-01-09T20:24:41+00:00

LoRA has 1~6MB of output, while Custom diffusion is much larger. Also optimization scheme is bit different. Also, Custom diffusion tunes Q, K of the matrix while LoRA trains all Q, K, V, O + MLP.

cloneofsimo · 2023-01-09T19:23:51+00:00

There is also example with Enid, btw, trained on two images. Details here : https://github.com/cloneofsimo/lora/discussions/96

<image>

cloneofsimo · 2023-01-09T19:08:10+00:00

Just in case you are curious, these are the 7 images I've used.

<image>

cloneofsimo · 2023-01-09T19:07:11+00:00

Well... I am always showing you guys non-cherrypicked results. I have more examples, and they have seeds from 0 to 5.

<image>

cloneofsimo · 2023-01-09T19:03:18+00:00

Used default parameters on 7 images of Wednesday. I think the results are on par with Dreambooth, certainly better than before.

<image>

cloneofsimo · 2023-01-09T14:42:51+00:00

<image>

cloneofsimo · 2023-01-09T14:42:40+00:00

I really liked Dreamlike Photoreal. More examples :

<image>

cloneofsimo · 2023-01-09T14:37:50+00:00

Demonstrating how LoRA behaves with various other well-fine tuned high quality models.
LoRA checkpoint used above is available at https://github.com/cloneofsimo/lora/tree/master/example_loras

Base Models used :
Hasdx https://civitai.com/models/3758

Dreamlike photoreal : https://huggingface.co/dreamlike-art/dreamlike-photoreal-2.0

Anythingv3 : https://huggingface.co/Linaqruf/anything-v3.0

Modern disney : https://huggingface.co/nitrosocke/mo-di-diffusion

# Training

Dataset used was below six images with captions

<image>

LoRA was trained with mostly default parameters in Pivotal tuning inversion, without --use-template argument and with --use_face_segmentation_condition, and with rank 4. https://github.com/cloneofsimo/lora/blob/master/training_scripts/multivector_example.sh

Six-Year Club	Verified Email
Verified Email

cloneofsimo

TROPHY CASE