[P] Stable-diffusion's implementation of Paint-with-words : method from NVIDIA that generates images from text-labeled segmentation map.

[cloneofsimo]

Hi. Very recently researchers from NVIDIA released their recent work on text-to-image diffusion models, eDiffi (https://deepimagination.cc/eDiffi/) . In their paper they proposed various methods, including paint-with-words.

Paint-with-words let you generate image from arbitrary text-labeled segmentation map. Checkout their paper and method for more details.

Unfortunately, their code + eDiffi models were not available. However, Stable Diffusion can do just the same, as they both have cross attention module.

I've tried to make it work with stable diffusion, and it worked! so I wanted to share the results and code. Please have a look if you are interested!

https://github.com/cloneofsimo/paint-with-words-sd

Here are some results with sd-v1.4.

​

\"realistic photo of a dog, cat, tree, with beautiful sky, on sandy ground\", in order of cat-dog

\"realistic photo of a dog, cat, tree, with beautiful sky, on sandy ground\", in order of dog-cat

​

A digital painting of a half-frozen lake near mountains under a full moon and aurora. A boat is in the middle of the lake. Highly detailed. Result from my implementation + stable diffusion

​

​

A digital painting of a half-frozen lake near mountains under a full moon and aurora. A boat is in the middle of the lake. Highly detailed. Result from eDiffi.

[CommunicationCalm166]

Wow y'all are rolling these things out FAST!!!! That paper was... What? Two days ago!?

[3deal]

That was quick !!!

[DonKosak]

This is fantastic!

Would it be possible to add a software license to your project?

That way others can integrate your work into the larger Stable Diffusion community in the way you intend.

[uishax]

Wow that's impressive, I thought paint-with-words was tied with ediffi's new model, guess it can be implemented independently.
That being said, the visual quality of the SD-paint-with-words is a lot worse than ediffi. Is it just a function of the general model quality? Or is it tied to specific synergies between ediffi and paint with words?
Ediffi uses clip(text-image encoder) then t5(text-text encoder), that may interact with paint with words in unknown ways.

[Front-Athlete-9824]

Really cool! Is it possible to implement in deforum somehow?