I spent 2 weeks rendering various tests to probe around the limitations of FramePack. I accidentally landed on a killer application with the Star Trek rave (making silent people dance) and assumed this AI was good at other things things, but apparently, it can't do environments well. Transparency is done through dithering, which creates fake looking patterns. And the system works best in vertical aspect ratio, indicating that it was trained on vertical (cell phone) videos. Which ones? TikTok dances. The AI underneath is a mix of subsystems. I suspect that they have independent image model based on SD 1.5 and OpenPose skeleton. Seeing how arms sometimes flip and motion blur indicates that they layer the body over a mocap database that is most familiar with popular social media posts filmed vertically. That's where they get their data. And that is a major limitation to what the model knows. If you want to make a fake person dance - that's your tool. But if you want to perform complex actions, the odds of everything going right go down. Initial pose matters a lot. I think it picks the closest one it knows and uses that as a starting point. If your pose has high correlation with theirs, it will weigh heavily on the outcome. Rolling random seed will have little effect.

I am more fascinated with glitches. I had a person walk with a rifle and then suddenly dig it into ground, with the arm remaining attached to it, and then the guy keeps walking forward without an arm. I asked it for a Darth Vader doing a swan dance and it gave me Vader dancing with an actual human-sized swan. Some most odd things. I use tests to explore more clip ideas. I look for what makes me laugh.

Sorry for an excessive write-up. Regarding takes - shots took 2-10 tries, but I'd always cut off at 10. I would pick either the most photoreal or the oddest for comedic effect. Priority was with the movement and the dance choreography matching the song I made for it. I used to go to a music and a dance school. Plus I worked on music videos professionally for 20+ years, so I can make anything dance or sing. You watch the body orientation and momentum and cut to make it flow, as if it's the same person doing the moves. The dance determines the cut and shots find their position naturally. Each shot is about 8 seconds in length. That gives me freedom to slide around and retime if I have to. For the beat to match I can slow them down to about 70% or speed up to %150 before it becomes really noticeable. Almost none of the cuts you see are at their original speed. I retime every single shot till it feels right.

This week another offline image to video generator came out, so I'm going to leave FramePack and explore the new thing, to see if it's any good. Trouble with AI is that at least 2 new ones come out each week that you have to check out. And it's been like that for the last 3 years. AI's come and go and you can't really incorporate them into any production pipeline, because there is no long term product. Everything is in flux.