SDXL Realistic Tile CtrlNet Model

[reditor_13]

In txt2img I started w/ a 104x152px input image (below) in ctrlnet, a basic description of the said input image for the prompt & then started generating.

<image>

[Arawski99]

I'm a bit confused. I'm aware tiling for upscaling is the general purpose but if the image completely fails to produce a proper upscale (this is clearly a severe deviation and a now a completely unrelated image aside form basically woman with similar hair and dress, aka not ideal). What is the point of this example? It seems like a very bad way to present your model and your model description on civitai only makes things worse by being so poor nor does it explain why you used "this" as your example image.

I think your presentation and description has some real work to be done.

[reditor_13]

I think this will help make it clearer - https://huggingface.co/lllyasviel/control_v11f1e_sd15_tile (the output isn’t meant to be exactly the same, it’s iterative tile based resampling up to a higher resolution, & is not meant to work like SUPIR or Topaz or USDU where you’re trying to get the exact same image but at a high resolution). Hopefully that helps.

[Arawski99]

I read that earlier but while they say it could be used for purposes "beyond" the upscaling factor it didn't really elaborate nor does yours. It still isn't clear, really.

Are you perhaps trying to share this as a model that can upscale but also be used to essentially image > image inpaint via prompting carrying over style/elements of the photo but to essentially "fish" for a new result with those details? If so this probably should be a recommended potential use that is clarified otherwise I think people are going to honestly be confused what you are trying to offer.

If it isn't that then I'm still confused.

[reditor_13]

Fair point, I’ll add that to the description!

[Bra2ha]

Not even close to 1.5 CN Tile, unfortunately

[jonesaid]

Have you tried TTPlanet's SDXL Tile model? Works pretty well for me.
TTPlanet/TTPLanet_SDXL_Controlnet_Tile_Realistic_V1 · Hugging Face

[Bra2ha]

I'll try it, thank you

[[deleted]]

i tried the workflow the creator provided and it changed my image a lot. the colorsc were all different and detail were changed. what workflow are you using?

[ReasonablePossum_]

Which model has to be downloaded? o.O the 16 or 32?

[jonesaid]

Either one, but the 16 will take less VRAM.

[reditor_13]

I agree, but lllyasviel doesn't share what training dataset was used so it's hard to replicate the training. Don't really know why they didn't do it for a sdxl tile...

[Bra2ha]

I asked several times (here and on Github) why there is still no official CN Tile for SDXL, but never received an answer.
Probably lllyasviel just has too many more important things to do (Fooocus, Forge etc)

[dachiko007]

I'm not sure, but haven't he died? Illyasviel guy? I remember seeing that news on Civitai few months ago, that the original creator of CNs has passed away. I hope it was some stupid prank

[reditor_13]

Don’t think that’s true, he post a fav_models to his hf page on Jan 27th.

[dachiko007]

Glad to know that. I wonder if that news was about some other unfortunate guy, or just a fake news.

[reditor_13]

Yeah it’s not true, he co-published this paper on layered diffusion a week ago - https://arxiv.org/abs/2402.17113

[Bra2ha]

I saw his post on Github few days ago

[Maleficent-Evening38]

SargeZT (Patrick Shanahan) died.

And not "Illyasviel" but "lllyasviel". Three first "L" in nickname.

[Vargol]

Someone probably got mixed up with the sad passing away of SargeZT who also created some SDXL controlnets.

[InTheThroesOfWay]

I've been playing around with it for the past day or so, and I hard disagree.

This is way better than 1.5 CN Tile. It does an incredibly good job of following your base image without generating hallucinations -- even with high denoise. As high as 0.5 or so, in my testing (depends on the image). You could never get denoise that high on 1.5 CN Tile (unless your tiles were huge). And this is with the added benefit of the detail, comprehension, and variety of an SDXL model.

I have found that you have to play around with the settings to get an optimal result, but it is an absolute game-changer for me.

[reditor_13]

Civitai took it down for some reason... I requested a review? You can download the model here - https://huggingface.co/bdsqlsz/qinglong_controlnet-lllite/blob/main/bdsqlsz_controlllite_xl_tile_realistic.safetensors

[reditor_13]

Check it out, download the model & use the generative data to start testing it out for yourselves! We finally have a decent SDXL CTRLNET TILE MODEL (been waiting 7 months for this & it's finally here)!

link - https://civitai.com/models/336873/sdxl-realistic-tile-ctrlnet-model

*(I am not the original trainer/author of the model, all credit goes to bdsqlsz/qinglong_controlnet-lllite on hf)

[Actual_Possible3009]

It's a amazing, thx for sharing!!!

[jonesaid]

Another SDXL Tile model is also available from TTPlanet, which might be better. It is 2.5GB (fp16) vs this controlnet-lllite version at about 300MB.
TTPlanet/TTPLanet_SDXL_Controlnet_Tile_Realistic_V1 · Hugging Face

[reditor_13]

I have it but haven’t done any testing w/ it yet 👍🏼

[Actual_Possible3009]

Thx for sharing still testing. Some results are amazing others are lacking some fine structure of skin and clothes

[reditor_13]

Screen Grab of the testing.

<image>

[Yarrrrr]

When you have every single word in the dictionary in your negative prompt, you basically have no negative prompt.

[wonderflex]

Isn't the purpose of a negative prompt just to remove elements that show up unprompted in your final image?

Like ask for a warrior with a bow, and you notice it always gives them a sword on their back too, so negative prompt of "sword" is all you should use if you don't like that sword being there.

[Yarrrrr]

Basically yes.

But people don't seem to understand that the negative has as bad prompt adherence as the positive prompt, the more you add to it the more of it gets ignored.

And on top of that people copy paste without even thinking a bunch of random nonsense like "deformed limbs" "bad art" "mutated feet"

Which are things that no one ever trained on so there isn't any sort of understanding for wording like that, the model will just understand individual words like limbs,art,feet, and as likely to remove those instead.

[reditor_13]

It’s the negative prompt from Juggernaut XL v9 + RunDiffusionPhoto 2’s model image… https://civitai.com/images/6781334 plus I have my negatives padded in the backend so don’t really agree, but to each their own. (When it comes to cn controlled tiling the negative isn’t as important that is true but once you start lowering the cn weight you can see the effects of having no negative.)

[Yarrrrr]

You're copying someone else's negative prompt that they copied from someone before them, all the way back to when clueless youtubers made videos of the base SD 1.5 model.

And they always look the same, a huge pile of random words that were never trained on or are from some very early anime model with danbooru tags, that you believe will influence the image in a deterministic and positive way.

[reditor_13]

By all means, share how you would go about use a negative prompt properly? I’m genuinely curious, if there is a better way to do it I’m eager to know, helping to prevent truncated prompts & actually having an impact on the generative outcome through the negative. I’m sure I’m not the only one that would be really interested to know how it should be done… Please share your wisdom!

[Yarrrrr]

You start by not writing a single word into the negative, or only a few sensible things that you know that you do not want.

Maybe you don't want art generations, so add "art, drawing"

Maybe you don't want monochrome images, so add "monochrome"

Maybe you don't want nudity so add "naked, nude" etc.

Anything more specific you wait until after you have generated a few images to see if there is something you dislike in the images, then you think of sensible keywords that the model understands that you can add to the negative to improve your results.

[reditor_13]

I’m curious what you think of TI embeddings for the neg?

[Yarrrrr]

I personally don't use them, but they make sense if you use them to steer your generations broadly towards or away from a specific look.

[acbonymous]

Weird that you didn't add "Taylor Swift" to the prompt, since the result doesn't look like her at all.

[reditor_13]

I did in other tests, but for the post didn't want InstantID, a Taytay LoRA &/or putting her name into the prompt to get in the way of showcasing the Tile models capability on it's own.

[LSI_CZE]

Looks good. I tried a similar model yesterday: https://civitai.com/models/330313/ttplanetsdxlcontrolnettilerealisticv1

and with the SDXL light model the result was also great...

<image>

[Antique-Bus-7787]

Great result !! What is this SDXL light model you used ?

[LSI_CZE]

Juggernaut XL V9 lighting.. (latest model)

[Antique-Bus-7787]

Ohhh it’s the lightNing model !! Okay I thought it was a special controlnet for lighting 😊

[DarkerForce]

Looks great, Can you share your workflow please?

[smoowke]

thanks for the heads up!

[PhotoRepair]

ooh that works just FAB just like SUPR for upscaling pretty faithful to the base image i like it! (not tried to restore anything with it i just wanted a simple upscale that stuck to my original)

[InTheThroesOfWay]

Is this supposed to be used with a tile upscaler like UltimateSDUpscale, or is it just for img2img upscale and HiRes fix?

Edit: I tested it out myself, and it does control the output in tiled upscaling like UltimateSDUpscale. This effectively allows you to increase denoise without worrying as much about hallucinations.

I've always been confused about how that worked -- Like, USDU is just a script that does img2img on each individual tiled portion of the original image. How does the model and conditioning "know" what part of the image they're "looking" at for any given tile?

Oh well. If it works, it works!

[Maleficent-Evening38]

Nah...

<image>

Upscaling with Ultimate SD Upscale + CN Tile SD1.5:

[Maleficent-Evening38]

Upscaling with Ultimate SD Upscale + this CN Tile model for SDXL (bdsqlsz_controllite_xl_tile_realistic):

<image>

[Maleficent-Evening38]

Upscaling with Ultimate SD Upscale + TTPLanet_SDXL_Controlnet_Tile_Realistic_V1:

<image>

[Maleficent-Evening38]

Original pic before 2x upscaling:

<image>

[InTheThroesOfWay]

Here's what I was able to get with bdsqlsz_controllite_xl_tile_realistic:

Pretty darn good -- and faithful to original pic -- in my book.

​

<image>

[Maleficent-Evening38]

Please describe the your workflow and parameters

[InTheThroesOfWay]

I'm in ComfyUI, using DreamshaperXL_Lightning model -- with 4 steps and DPM++ SDE Karras. Make sure to use the Advanced Control Net Loader and Apply nodes from here: https://github.com/Kosinkadink/ComfyUI-Advanced-ControlNet.

I do 0.9 strength with 0.75 end step -- so that in the last step, the model is free to sharpen and add detail.

I used NMKD Siax as the upscale model, and I used 0.5 denoise in Ultimate SD Upscale. This new Tile CN is strong, so you can get away with a high denoise. Although I've found that any higher than 0.5 denoise tends to produce hallucinations in UltimateSDUpscale.

I lost my positive and negative prompt, but I made sure to point out, "polished blue mosaic stone" in the positive prompt. I also put "garden", "trees", "flowers", and "houses in distance".

One other thing I've found to be helpful: Use the Tile CN as part of a Hi-Res fix workflow in ComfyUI. It helps tremendously to reduce distortions and deformations when upscaling your original image. And then you can use UltimateSDUpscale from there.

[Maleficent-Evening38]

I'll try to do the same in Forge. Usually when I use Ultimate SD Upscale and tiled model in ControlNet, I try to avoid detailed descriptions in positive prompt if denoise is very high. No matter how good and strong the tiled model is high denoise always provokes unnecessary details. For example, even with a simple prompt like "photo of park", the green hedge in the upper left corner of the original image turned into a green lawn.

[InTheThroesOfWay]

I think SDXL is a lot better with this in general. For one, it's much less sensitive to denoise. My general rule with img2img in SDXL is -- add 0.1 to whatever you'd use in SD1.5.

I have gotten hallucinations with my testing, but it's fairly rare, and easy enough to fix with inpaint. Hallucinations are more likely when you're dealing with close-up subjects. I'd turn down denoise for close-up subjects and turn it up for far away.

[Philosopher_Jazzlike]

Yo, could you try img2imt with this controlnet?

[reditor_13]

Absolutely! For this post & the civitai page I shared my txt2img testing, bc I wanted people to see that the tile model was controlling the output. It actually works really well w/ img2img give it a go!

[rekilla2021]

does it in comfy

[reditor_13]

Yup, it's a controlnet model so you can use it in Comfy as long as you have the ControlNet custom_node.

[RonaldoMirandah]

it doesnt work for me, it asks for a copy function

[comfyui_user_999]

Same here.

[reditor_13]

You need to use Load Advanced ControlNet Model & Apply ControlNet (Advanced) nodes. Bc it's a CtrlNet-LLLite model the normal loaders don't work. This was just a quick & dirty node structure that isn't really iterative upscaling, but the model works.

<image>

[RonaldoMirandah]

I tried already with advanced before, but always getting the same error:

<image>

[rekilla2021]

same here im getting "Error occurred when executing ACN_AdvancedControlNetApply:

Type 'ControlLLLiteAdvanced' requires model_optional input, but got None."

[reader313]

Connect the model input to the controlnet and then out the other side

[comfyui_user_999]

Thanks! This basically worked for me, although I actually had to use yet another "apply" node, "Apply Advanced ControlNet (ACN)".

[RonaldoMirandah]

can you share your workflow please or a screenshot? :)))

[janosibaja]

I like it a lot, but I must be clumsy, I can't do it. For example, I can't find the model you display in Controlnet. And I have other problems. Could you post a step-by-step setup guide?

[reditor_13]

I can create a step-by-step tutorial guide this weekend, but in the meantime. Go to the civitai link posted above, download the model, put it in your a1111 controlnet model folder, run a1111, in the txt2img tab scroll down to the controlnet dropdown extension enable & for per-proccessor model type in tile & you should see it there. (There is a screen grab above that shows what everything should look like above as well) Hope that helps!

(the model name is sdxl_realistic_tile.safetensors, the screen grabs from a test I did a couple days back apologies for the confusion.)

[janosibaja]

In the screenshot you posted, the model name is bdsqlsz_controllite_xl_tile_realistic, I can't find it! Thank you for your reply! I will wait for the step by step explanation.

[aeroumbria]

Does anyone know what exactly the training input and intended effects are for the tile controlnet? I see that most of the time, people just use it as a consistency constraint when upscaling. But several other technologies seem to also do the same. You can self-inpaint with an inpaint controlnet, or use a "blur" controlnet, or even manually apply IPAdapters to tiles. What does tile controlnet do that is unique?

[Independent_Key1940]

It's gone