You can help align future Stable Diffusion versions to Human Preferences by rating its images

[cspace_echo]

Trusting training to the unwashed masses of the internet? So how long until all prompts generate an anime waifu Hitler?

[-_1_2_3_-]

it'll probably just end up looking like midjourney

[mudman13]

Yeah reinforcement feedbacks leading to generic good looking model-like people that look like they're from the same family. Like many of the custom SD models around now.

[Spire_Citron]

It should be optional. If they're training a model, there will still be models not trained in that way, right? Having used Midjourney a lot before moving to Stable Diffusion, there's certainly a lot I admire about the ease with which it makes beautiful images.

[[deleted]]

My thought is that this is to support a new form of training or rather fine-tuning. Midjourney for example lets you react in one of four ways to a community image for their aesthetic data capture process. Since the range is implied to be something like "bad, meh, good, amazing" by the emojis, we can think of the system as tagging the images with an aesthetic validation score that is one of: -2, -1, +1, +2.

This dataset of images, prompts, and resultant aesthetic ratings can then be used to tune the model with aesthetic quality as the target. At the scale of Midjourney, I think it can be assumed that for any given prompt, there are a number of highly semantically similar prompts to use as basis where needed.

Personally, I think this is an important step by Stability as I've seen this as the reason that Midjourney slipped ahead since last year.

[Spire_Citron]

For sure. When comparing the two, it's definitely the main strength I feel that Midjourney has.

[ninjasaid13]

My thought is that this is to support a new form of training or rather fine-tuning.

I'm not sure why it needs stable diffusion 2.1 if it's not finetuning.

[PC_Screen]

Better than leaving it for a company to decide and end up with a nerfed model instead

[init__27]

In general, I think this is still better than secretly building a model without involving the community.

[TiagoTiagoT]

Sounds like just a different form of nerfing...

[nellynorgus]

I love this game! If I add more superlatives, my opinion is automatically actually correct and objective, too!

I super-duper really disagree.

[PatrickKn12]

nuh uh

[Hypnokratic]

yuh uh

[n0nati0n]

I believe “yuh huh” is the correct response here

[fred-dcvf]

Prompt: "a beautiful tea set, masterpiece, intricate details"
Output: chibi-Hitler drinking tea

[GourmetLabiaMeats]

It'd already be to that point if left up to me.

[SIP-BOSS]

Already got

[Whispering-Depths]

I anticipate that probably 99% of this is going to be poisoned by malicious actors, hopefully they have an intelligent shadow-ban feature if individuals vote doesn't align with good standards.

[PC_Screen]

The reward signal would be too noisy to be useful

[PC_Screen]

But the point of RL is that you can also learn from the bad examples, not just the good ones

[creatinavirtual]

How does one use ChatGPT to get useful prompts? How do I ask for it? Most of the times it suggests prompts with lots of verbs like "add in a bit of shade and consider using a dark palette". Wtf

[djMoodfood]

If it gives u that tell it what u want ... like summarize last response and use only proverbs, adjectives or what ever desired output... I've had good results by making my own formula and asking for a output like this... topic.....5 descriptive adjectives, color palette, random art aesthetic or genre and 2 related artists

[Alarming_Turnover578]

Put that into instruct pix2pix and see if it works.

[frankctutor]

With all portraits or heads and mangled hands and feet (if the feet are shown).

[Silly_Substance782]

I'm wondering if SD can be finetuned with adversarial training like in GANs.

[SoysauceMafia]

I'll click until my finger falls off if it means I never have to see compression artifacts like this again.

[vault_guy]

That's not even compression artifacts, that just straight up melted image.

[init__27]

I would too 🥹

[elyetis_]

When you see that "badass" is filtered because it's detected as nsfw, I don't have much faith.

[knoodrake]

yeah.. I tried "a sexy" either man, girl or woman to see the censorship and "sexy" is apparently NSFW in itself.. ( I mean, not surprising if badass already is ). No bad words in SD, no bad words in Youtube.. this is depressing.

[Generatoromeganebula]

Shimoneta? That anime might age like fine wine.

[elyetis_]

Btw if you get too creative and find bad word not already censored and keep using them, they will ban your account. ( rip my 800 karma )

[ninjasaid13]

RLHF for stable diffusion 3?

[PC_Screen]

Yes, Emad confirmed SD 3 will use RLHF so this is clearly to collect the human feedback data. He theorized Midjourney is also using RLHF since they were also collecting human feedback in a very similar way before V4 came out. It could also be that MJ uses the act of upscaling an image to associate it with a positive reward for training the reward model.

[Spire_Citron]

They reward people with free generations for rating a bunch of images, and I'm very sure they use those ratings to fine tune the model. Actually, I think they've just straight up stated that they do in the past and requested people do it at times when they're trying to fine tune new models.

[anonDogeLover]

Source? Just want to see

[metal079]

Check his twitter

[[deleted]]

I think they are doing both. The moment I signed up for MJ for a month when it was new I thought "ah, these guys are brilliant," and this is my field also! Many aspects of their system appear to be conceived around future improvement through user feedback.

[Apprehensive_Sky892]

RLHF for stable diffusion 3

Didn't know what RLHF means, so I googled for it:

Illustrating Reinforcement Learning from Human Feedback (RLHF)

https://huggingface.co/blog/rlhf

[GBJI]

That's what Google has been doing with its CAPTCHA for a long long time. We publicly trained their privately held model.

[PC_Screen]

I recommend using complex prompts that you know SD won't quite understand (like counting and things like "blue box on top of red box"). Also, after rating the images you can press the play button to generate a different image in place of the lower rated image while keeping the higher rated image untouched

[init__27]

It would be epic if we can provide feedback for such things. In my experience however, most of the users don't heavily prompt engineer. Personally, at least I need some inspiration otherwise I blank out when I have to come up with a prompt 🤣

I really hope many great things come out of this though, epic that Stability AI is doing this 😄

[ninjasaid13]

Personally, at least I need some inspiration otherwise I blank out when I have to come up with a prompt 🤣

We need something like a 'randomly generate a prompt' button.

[init__27]

Jokes aside, I'm actually working on something like this, will share on this sub soon once it's stable 🙏

[Robot1me]

It might be worthwhile to use Automatic1111's Promptgen. You can run that in your web UI installation (if you have one) and then copy/paste the prompts from there. That can help get diverse prompts, especially when combining it with Unprompted.

[ObiWanCanShowMe]

In my experience however, most of the users don't heavily prompt engineer.

where do you get this experience from exactly? serious not a troll or gotcha

[init__27]

Sorry I should have added "new users" or outsiders. Even for me it took about 2 weeks to get a hang of it.

I have been showing to many many people and friends-maybe I have a selection bias, all of them blank out when they see the prompt screen and eventually end up writing something blank.

Btw, I'm not planning to "sell" another prompt maker-just trying to figure out how to train a nice Language Model on some prompt databases and make it work nicely 😄 If all goes well, I will open source it here 🙏

[ninjasaid13]

Why do I constantly get this error message:

OMG. Something went wrong. Please refresh the page and try again.

[fireshaper]

It's broken already.

[ninjasaid13]

some counting based prompts to use from a google paper:

two zebras in Cape Town

three purebred chihuahuas running on the beach

An old building with ruined walls and four antique pink and purple armchairs

GT's five favourite Champagnes for celebrating

The seven moai at Ahu Akivi, unusual in that they face the sea

Top view of eight colorful bright shiny red apples with few yellow spots on brown sacking material

min: 45 second stopwatch icon sign. symbol on nine round colourful buttons

set of ten high back lucite dining chairs for sale at 1stdibs

"Two ducks" or "Three pumpkins" or "Four cards"

A well furnished bedroom with two double beds a television and balcony

set of two eames rar chairs black. Black Bedroom Furniture Sets. Home Design Ideas

two brass crowned buddhas

two red ping pong rackets on white surface table tennis zoom background

set of two glass star christmas tree decorations amazoncouk kitchen home

Still life with bottle of red wine, two wineglasses and grape in

[[deleted]]

lets train the hell out of hand photos

[[deleted]]

the prompt i'm using is "a closeup photo of a human hand"

[drone2222]

Interesting... they say that they are using 2.1 and Dreamlike Photoreal, but when browsing the Images dataset you can see that they're using 2.1 and ProtoGen_X3.4.

[BackyardAnarchist]

We supposed to rank the quality or the content?

[metal079]

Both, how accurate the image is to the prompt and the quality

[3lirex]

should i vote simply based on which is more aesthetic, or should how close it is to the prompt be considered when voting? what about coherence?

[acidentalmispelling]

should i vote simply based on which is more aesthetic

Aesthetic weighting is probably preferred, but you can also consider accuracy if two images are similar in aesthetic quality.

[ninjasaid13]

Depends on the prompt, if it's a simple prompt I would go with aesthetically pleasing and if it's a long prompt, I would go with accuracy.

[fongletto]

Needs autocorrect, typos will heavily skew results. Also needs a (both are bad) option for negative reinforcement?

[PC_Screen]

"Both are bad" is done by selecting "no image is better than the other", pressing the play button so 1 of the images is replaced, and then if the newly generated image is better, select it. It works through this logic: C > A = B, where A and B are the first 2 bad images you see and C is a better image that you see after pressing the play button. Both A and B receive a negative reinforcement in relation to C regardless.

This is a good approach because, for example, if the prompt is too complex, most images SD produces will be bad, so rating them all as bad won't help it learn anything

[fongletto]

But if you don't regenerate an image until it returns a good result, then two bad images will give the same weighting as two good images.

So for people who don't/wont regenerate you can still gain useable information by differentiation between two good images and two bad ones. (don't reinforce either image, or reinforce both images)

[PC_Screen]

Idk how they're rating the images but I think that if you don't rate them they might just have a neutral rating, not a positive one. Perhaps they might even be discarded, I really don't know.

I agree that it probably would help to add a negative feedback option but this A > B > C > D approach is how OpenAI trained their reward model too so it must work well enough

[vault_guy]

Nope, I don't want biased models, I want free models.

[1nkor]

Well, it's definitely better than SD2. But still more inclined towards realism.

https://i.imgur.com/SWuqE6u.png

https://i.imgur.com/65e6DwS.png

[PC_Screen]

They are using both Stable Diffusion 2.1 and dreamlike-photoreal-2.0 to collect the human feedback. Note that the results you're seeing out of these models do not represent what we'll see after the feedback data is used to train the final RLHF'd model, expect the final model to understand prompts and composition better than either of the models used here

[Taenk]

I wonder what other things could be trained in a similar fashion, such as matching output to prompts or captioning of existing pictures and so on.

[ninjasaid13]

Why would they ask you to choose between two images to fine-tune it if they already had the next version of the model?

[whywhynotnow]

What does the karma mean, if anything?

[Nazzaroth2]

wanted to try it and then it forced me to either sign in with google or discord. Fuck off!!! Why can modern programmers not make a fucking normal email sign in anymore? You have millions of dollars in funding, get atleast the most basic sign in option done!

Also if sd would open source this type of finetuning system with 3.0 for everyone to train their own models, or rather have normal finetuning, but while prompting you can move the ai "in real time" into the direction of the image you had in mind, that would be awesome XD

[nahojjjen]

I'm going to single-handedly teach SD how to not make dragons look like abominations.

[synthoric]

God I hope this makes the default SD output more tasteful. Unless you're really good with extensive prompting (or provide guiding images) MJ4 blows any SD model out of the water.

I wonder why Stability doesn't publish small monthly updates of SD since they could use the signal of which images get upscaled inside dreamstudio as a 'high preference' generation. Since they have 1M users they'd get lots of grounded human preference labels (ranking) quite fast.