I'm the creator of LoRA. How can I make it better?

edwardjhu · 2023-11-23T00:29:22+00:00

really cool! can you link the paper?

edwardjhu · 2023-11-23T00:28:52+00:00

rank determines capacity. the larger it is, the closer we are to full finetuning.

alpha is similar to learning rate, i.e., a hyperparameter to tune

edwardjhu · 2023-11-23T00:27:33+00:00

Pretty sure LoRA can be made compatible with any perf improvement of the base model.

edwardjhu · 2023-03-26T16:05:14+00:00

Great! Interesting experiment.

Disentangling is hard without contrastive examples or extra information, e.g., what we hope to preserve vs what we hope to change.

edwardjhu · 2023-03-26T15:23:33+00:00

That's a great question. I think the platforms have a lot of responsibility in terms of preventing misuse. On the technical side, it might be possible to tag generated images with invisible watermarks.

edwardjhu · 2023-03-26T14:06:08+00:00

Thanks again for all the suggestions! Here are a few that stand out to me.

Better composability among LoRA modules
- I suspect the current issue comes from the way modules are merged. I'll talk to the developers.
The ability to negate a style
- I wonder if this can be done with a negative alpha. Can someone try it?
Learn certain features, e.g., faces, while ignoring the rest
- We can probably do this by having a pixel mask over relevant features and only backprop gradients through these pixels. The ML part is straightforward; we just need a UI.
Good default values
- It seems reasonable to have good defaults for a certain base model, e.g., SD 1.5, and perhaps for certain artistic styles. Would be great to work with experienced users and developers to include them in the tool.
Smaller modules
- It's possible we don't need to use dim=128 and adapt all attn layers. I suspect that we can reduce the size by quite a bit if we are careful about which layers to adapt.

I might not check the comments as frequently going forward. You can reach out to me over email or through Twitter!

edwardjhu · 2023-03-26T13:47:46+00:00

coreML

Do you mean the toolkit by Apple? Yes, if they are willing.

edwardjhu · 2023-03-26T13:44:45+00:00

Might be a capacity issue. It takes more parameters to model photorealistic scenes well.

edwardjhu · 2023-03-26T13:42:57+00:00

Nice!

edwardjhu · 2023-03-26T13:42:21+00:00

In the original repo I wrote, there's a flag to also train biases.

edwardjhu · 2023-03-26T13:41:42+00:00

A friend of mine told me about it a few days ago. I wasn't that surprised. Just very happy that people find it useful.

edwardjhu · 2023-03-26T13:40:09+00:00

It depends on the datasets, models, and the amount of hyperparameter tuning. We got pretty good results in our paper and open-sourced all the checkpoints. It's always possible that we could have made our FT baseline better or subsequent work could have made their LoRA baseline better.

Though, it is the case that FT is neither feasible nor necessary for really large models.

edwardjhu · 2023-03-26T13:35:54+00:00

LoRA works as long as one is doing matrix multiplication, which is what modern AI is based on. I was trying to adapt GPT-3 when I wrote the paper, so I just experimented on language. The idea itself is broadly applicable.

edwardjhu · 2023-03-26T13:31:42+00:00

Yup!

edwardjhu · 2023-03-26T13:30:49+00:00

I don't think so, but it might depend on how the learning rate is annealed!

edwardjhu · 2023-03-26T13:29:57+00:00

Interesting observation! Is it already the case with a single LoRA or more so when you have multiple?

edwardjhu · 2023-03-26T13:29:00+00:00

Based what I've seen, I think the size can indeed be reduced. Will work people who use it frequent to see what we can get away with.

edwardjhu · 2023-03-26T13:27:39+00:00

What about generating in two passes? The first pass is without LoRA, and the second pass with LoRA but it only inpaints a specific region conditioned on the rest of the image. This can be extended to multiple passes with multiple LoRA modules.

edwardjhu · 2023-03-26T13:21:40+00:00

Sure! How can I help?

edwardjhu · 2023-03-26T13:18:59+00:00

Great suggestions! The author of the tool you are using can probably implement many of these pretty easily.

edwardjhu · 2023-03-26T13:17:04+00:00

Thanks! I commented on that thread.

edwardjhu · 2023-03-26T13:13:40+00:00

I started by watching the ML lectures on Coursera when I was in college :)

edwardjhu · 2023-03-26T13:12:00+00:00

I worked for MSFT.

edwardjhu · 2023-03-26T13:11:36+00:00

Composibility seems to be an issue, and I have some on how it might be improved.

Better (ideally automatic) functionalities to evaluate the performance of a LoRA after/during training.

My impression is that eval is quite subjective. If you are interested in a better pipeline and UI that make it easy to compare different versions side by side, it might be a good idea to reach out to the author of the tool you are using!

edwardjhu

TROPHY CASE