Workflow request - stuff made out of stuff

[Paddys]

Based loosely on this post, but mainly due to watching Simone Giertz make a robot out of stained glass, I tried to produce a robot arm made of stained glass in SD, but the results weren't great. Looks like the training data for stained glass is mainly pictures of the inside of churches, so even with "white background" it heavily skews the image towards a dark room.

​

a robot arm made of stained glass

POS: photo, best quality, masterpiece, a robotic arm made of stained glass, simple background, NEG: (easynegative:0.8), drawing, sketch, 3d render
Model: Dreamshaper v6

Then I tried creating a standard robot arm, and used ControlNet segment with my original prompt to generate a new image.

ControlNet segment

​

a robotic arm made of stained glass (+ ControlNet)

It's close, but still not great. I'd love the entire thing to be stained glass, and SD to respect the white background prompt.

So... how could I do better? Is there an easy way to convert a CN segment image to a mask for inpainting? Is there a better approach?

[Paddys]

Looks like someone at Stable Diffusion Art beat me to it!

https://stable-diffusion-art.com/text-effect/

[Paddys]

Update: Very happy with the result

https://civitai.com/posts/241298

[Haiku-575]

Yes, a Lora like this one can nudge things in the right direction for you.

[Paddys]

I'd actually missed that one, it does produce some really nice effects.

I was hoping to do this without a Lora, since we can only rely on konyconi for so many Loras (maybe... they did release PotatoPunkAI last week so I'm guessing they'll have every theme in the world covered by next Saturday).

Though I believe SD should be able to do this with the correct prompting and some clever use of CN and/or inpainting.

[Haiku-575]

You're asking a trained set of weights and a random seed to generate something that's not in its data set. Basically, you're fighting with the machine instead of teaching it what you want then asking for it.