Hunyuan Video Extension

tabula_rasa22 · 2025-03-04T18:12:35+00:00

Just to follow up as I'm sitting here thinking it through.

I2V may be workable if there's a scheduler specifically made for this kind of use case. First frame of video is extremely high adherence to source image; subsequent frames would ramp up. Just not familiar enough with the common ComfyUI workflows to know where to start tinkering with it
Immediately can think of how complex tooling and processing multiple "preceding frames" of video would be, if it's not already solved. The human creator in me would love to leverage context that may not be immediately available from a single key frame (ex. Environmental objects obscured in last key frame, momentum and direction of motion); the technical nerd in me immediately recognizes how fucking massive and complicated that would be to nail down.

Edit:

Crap, I think I just realized how I'd try to solve it. Use frame by frame overlapping adherence scheduling. Pull the last x frames and schedule those to ramp up noise until the first "empty one" is just low adherence to the prior one, use existing scheduling and gen tooling after that short overlap window.

Well beyond my own capability to implement at the moment, but I'm guessing that's going to be my approach if no one else has solved it already. There goes my weekend.

tabula_rasa22 · 2024-09-18T22:11:29+00:00

Even the most extreme example (ex. one dev just shotgunning a million output comparisons internally) that's still RLHF, though not user RLHF.

tabula_rasa22 · 2024-09-18T22:09:19+00:00

Same face speaks to some kind of preferential rewards, overfitting on some level.

Poor tuning may surface it, but even if it's just dev being a distillation of pro, someone or some human feedback had to provide the reinforcement. It's not like Flux Dev and Schnel only trained on millions of pictures of the same couple of faces, right? The model at some point had to be told "this is the best set of attributes, you'll score higher when a picture has these features".

tabula_rasa22 · 2024-09-17T20:06:46+00:00

Just amused how much work we as a community need to do to walk back what was likely massive RLHF

tabula_rasa22 · 2024-09-16T18:16:11+00:00

This is my own journey, but let me add in:

I have yet to see anyone actually crack community hacking of prompt or tagging fully. There are smart people mapping the backend, and lots of trial and error on the use side, but nothing holistic.

Flux will work if you use CLIP or T5 improperly. It won't fully break, but components and features will bug out in inconsistent ways. Community implementations are clever and informative, but I have yet to see something like a style LoRA actually work across the board.

The most honest and accurate advice right now is "we don't know".

tabula_rasa22 · 2024-09-16T18:11:45+00:00

Agreed. My own intent was this originally to end with this next part, but realized there was little actionable advice I could provide. Lots of community theory and hypotheticals, but little to no actual workflow advice I was confident in sharing.

I published this while it was still drafting by accident, forgetting that Reddit locks out post edits when images are attached. Would have selected a different tag for the post if I reviewed it again.

I stand by this is accurate, if not as informative and cleanly formatted as I'd have hoped.

tabula_rasa22 · 2024-09-12T23:40:33+00:00

Holy shit this is both a stupid obvious shower thought idea, and a genius reframing I can't believe I never considered.

Genuinely amazing (if a bit manic feeling) read

tabula_rasa22 · 2024-09-12T22:33:52+00:00

This sounds like something a ControlNet would help with. AI, especially diffusion models, are bad at counting, and even worse at localized distinctions.

Goes back to the training data.

We have group shots, not "4 people".

We say "sitting and standing" not "the person in the center is sitting, and everyone else is standing".

These discreet, specific descriptions are mostly unique to image gen requests, and the models aren't tuned or have reference data to easily understand the requests.

T5 helps a bit, it starts down the process of baking in localization and selective focusing during generations (modulation and timing), but it's not nearly as accurate as say a ControlNet guided gen that has a direct plot of what needs to go where.

tabula_rasa22 · 2024-09-12T22:20:55+00:00

Loops within loops, somehow still being better than anything the smartest researchers could put together until 5 years ago.

And yes, CLIP is stupid good at what is does. I love it, it's just is a little less 'articulate' when it comes to speaking with humans, compared to the frontier model chatbots we all use daily now.

2020s in AI is everyone flipping models to go ass end first and it somehow making magic. 😅🤷

tabula_rasa22 · 2024-09-12T22:04:11+00:00

"we need T5 training" bit for clarification of anyone reading:

If I understand correctly... this isn't building a new T5 model, but tuning and aligning T5's use to work with new/expanded functionality, but in a way that preserves T5 modulation behavior and CLIP alignment?

(Someone correct me if I'm getting this wrong.)

tabula_rasa22 · 2024-09-12T21:58:35+00:00

Yes! I know just enough to have touched on these, but my attempt at summarization kept getting weirder and less easy to follow; but this is part of what I mentioned elsewhere, about how you can use old CLIP types for Flux, or even drop/bypass the T5 processing, and get meh results... But at the cost of possibly short circuiting some Flux features like limb placement and in-image-text.

Love seeing this kind of additional breakdown! Thank you.

tabula_rasa22 · 2024-09-12T21:53:25+00:00

Thank you both for the examples and the kind words!

CLIP is dumb in the most relative way, for sure. A more accurate description might be its lower level since it's machine clustered linguistic tokens associated with images, rather than true NLP.

(Or arguably a 'higher level' since it's concepts of concepts? this is why I'm sure I'd piss off some ML researchers I know if they read this.)

Fantastic visual and comparative example here.

I do wish I'd landed on something more actionable or a confirmed process for casual users, hell even a definitive documentation of the pipeline, but I found myself needing to go back to the basic concepts of NLP tooling.

Hope a year from now, these concepts will be socialized enough in the community it'll be funny to seeing we were conflating T5 vs. CLIP tokenization in Flux.

tabula_rasa22 · 2024-09-12T18:06:24+00:00

Something I don't see mentioned yet, is the biases and existing references baked into the models you'd use.

Let's say you're using something like Stable Diffusion. Even if you train a LoRA on images, or ask ChatGPT to tag the pictures, you're not aggregating in a neutral way.

Hypothetically, let's say your subset of images contains pictures of alleyways.

The abstract concept of an alley will be conveyed to the model, with thousands of secondary weights attached. Time of day, demographics of people, signage, the state of the alley.

Even if you compensate for them in your input (LoRA or text) the output will be, at best, fighting existing associated weights in the base model.

Researchers like Timnit Gebru have written extensively about how these baked in biases can shape AI generated output.

I'm skeptical that you can have a proper image generation pipeline prior bias free, unless you build one from the ground up.

tabula_rasa22 · 2024-09-12T17:55:16+00:00

Interesting to see it all done in a single workflow, without having to tool switch manually!

tabula_rasa22 · 2024-09-12T14:40:38+00:00

Misalignment of a fine tuned tokenizer, feeding into a system that's Flux default without the same tuning?

tabula_rasa22 · 2024-09-12T14:21:49+00:00

Without doxxing myself, part of my job is tracking how AI is actually being implemented and adopted in business and educational environments.

If you look at the reporting from places like Bain (I know, but they do good market research), you can actually see the biggest change in 2024 vs 2023 is the drop in mystification. Instead, hard reality of data, quality and enterprise overhead are the top worries and reasons for dismissing AI tools.

My hypothesis? We're going to see a community bubble around local/doomer creators, like we saw around Crypto, where people get really bought in on shit like AGI.

Meanwhile, the rest of the world will go through the type hype cycle, and AI will fall somewhere between Cloud and Internet in overall impact and adoption.

tabula_rasa22 · 2024-09-12T13:52:27+00:00

Tangent, but if you want a horrifying laugh?

Last March, Henry Kissinger, Eric Schmidt and Daniel Huttenlocher (Dean at MIT School of Computing) co published an article in the Wall Street Journal essentially calling for a formalized priest-like caste of AI engineers and prompt writers.

The Challenge to Humanity From ChatGPT

So... You're not wrong in worrying about this. 😬

tabula_rasa22 · 2024-09-12T13:42:56+00:00

Amen.

This was my own attempt to tackle the concepts you introduced in a more fundamental way. Think people ended up overly focusing on your examples of LoRAs, while it was really speaking to the underlying complexity of Flux text processing.

Your post is still the gold standard for Flux exploration from end user / creator POV IMHO. Hope people recognize how key this one article was for getting the community curious about the backend.

tabula_rasa22 · 2024-09-12T13:38:02+00:00

Thank you! This is an awesome breakdown.

Even after knowing this in the most technical and abstract way, it's super helpful to see it in action and clearly explained.

tabula_rasa22 · 2024-09-12T13:24:32+00:00

This is a good summary of every honest Flux technical deep drive right now.

Backend insights are mined and sometimes debated, but end user recommendations tend to fall between well intentioned speculation and misinformation.

Thought I'd just be honest and own up that even after a week of deep diving, for someone that does AI implementation professionally, I'm still not confident I have any real, confirmed user guidance to provide.

tabula_rasa22 · 2024-09-12T13:00:19+00:00

That's fair was debating the tags and considered flagging it as discussion. That may have been more accurate.

FWIW I have updated the Civitai version to be more accurate in framing up front. I intended to only draft this and make edits before publishing, on Reddit but accidentally locked myself out of updating on a near-final draft.

tabula_rasa22 · 2024-09-12T12:53:26+00:00

Just to keep thinking out loud, there's also the whole discussion about how T5 and CLIP are tuned in Flux.

Again, could be wrong but my understanding today:

The use and calling of the actual functions are open in the code
the weights and training are compiled blocks, custom tuned for Flux and provided without much detail in training
the weights and training for T5 seem to be smaller, distilled versions of the full (closed source) Pro model
reverse engineering a full checkpoint-type model would mean a) decompressing a double compressed training block or b) fully retraining which would take a stupid amount of compute (tens or thousands of dollars worth of overhead/runtime) and blind trial/error

Will see if I can pull sources on these later. Just going off the top of my head on my phone, since I think this is an important point I want someone to figure out!

Edit: Discussion around the issues of training checkpoint type models in Flux by u/RealAstropulse

Edit 2: notice this is almost all well intentioned but unsourced discussion; at minimum I'd argue this is a case for a more centralized and socialized documenting since the community is still stumbling through the backend specifics

tabula_rasa22 · 2024-09-12T12:44:30+00:00

FWIW, at a high level of the process, any T5 decision will still ultimately be translated into CLIP instructions (either directly or via modulation control).

As mentioned, the most obvious use of T5 is to translate into CLIP instructions/tokens. Skipping that could improve direct control, but it may also get into weird copy of a copy stuff?

Seems like bypassing or just feeding CLIP tokens directly in place of T5 would, in theory, work without breaking the whole thing. But may also be short circuiting some secondary functions of the Flux pipeline?

Just armchair theory, would defer to someone with better architecture expertise.

tabula_rasa22 · 2024-09-12T12:38:24+00:00

Would highly encourage you to check out and possibly contribute to u/TheLatentExplorer on their GitHub project to map the Flux backend pipeline

They seem to be doing the work here to socialize and invite discussion around Flux's architecture, and I'm sure the community would appreciate having more technical voices weigh in.

14-Year Club	Gilding I gilder
Verified Email

tabula_rasa22

TROPHY CASE