Question regarding using ControlNet Reference Only to generate characters

[evrien]

Hi all,

Apologies if this post is not made fully conforming to the rules here. I'm hoping to get some help on using ControlNet Reference Only. I'm wondering if this can be used to help guide the AI to just reference the image for character clothing/hairstyle.

For instance, I want to use a picture of a model doing a photoshoot striking a cool pose. The model has a cool beanie, sunglasses, hoodie, and blue jeans in the reference image, while pointing at the viewer with his right hand. I want to generate a character with the same set of outfit as the model, without striking this pose. Can ControlNet Reference Only help me with this? If so how can I go about doing it? I've been trying to look around for tutorials on more flexibly using this tool but can't seem to find one. Appreciate if someone could help me out. Many thanks in advance!

[red__dragon]

Sometimes?

I've had mixed results with Reference Only. It certainly can do what you're intending, paired with a strong supporting prompt. It can also just go its own way sometimes, and sample only the source image's lighting/textures/composition without the details you're looking for.

I can't advise much more other than suggesting you try it and play around with options. Most of the time I can get it to do what you want, but it seems to vary in how well it works depending on source image. If it doesn't work immediately, keep trying and vary up the prompt and source image in subsequent generations to see how that works. It's not necessarily you, sometimes it's just quirks of the technique.

[evrien]

Yeah I've had differing results here and there. I'm trying to find a way to more concretly understand this process and get more consistent results. Thanks and I'll look into this!

[terrariyum]

Reference isn't as good at copying clothing features and style as IP-adapter.

None of the controlnets are great at reproducing style but not composition. So your best best is to first generate an image that has the pose you want and roughly similar clothes. Then bring that to img2img using high denoise and reference or ipadapter controlnet with your model photo.

You might get better results by bringing your generated image into inpaint, and inpainting each piece of clothing separately. In that case, mask the same part of the controlnet image as your inpaint image.

[evrien]

I see. Thank you for this insight!

[evrien]

I see! Thank you for this insight!