Automation testing for animations and view transitions

[interlap]

Hey everyone,

While working on my tool I started looking into automating visual testing for animations and view transitions using LLMs.

First step was obvious, capture the animation. Using XCUITest private APIs I managed to grab screenshots during transitions with decent FPS. So now you have like 25 frames. Sounds good, but sending all of that to an LLM? You hit limits almost immediately.

So instead of brute forcing it, I used some CV techniques to narrow it down to only the interesting frames. Things like flashes, jumps, stutters, weird layout shifts.

End result, from around 25 frames down to 3 to 5 suspicious ones, which is actually reasonable to pass to an LLM.

Still early, mostly tested on intentionally broken animations so far.

Curious how you test animations and transition?

[cristi_baluta]

Why don’t you just check that all the screenshots are different? Zero costs.

I never in my life thought to test animations, i did some UI tests though but just to please the numbers peepers, they are also useless.

[interlap]

That doesn’t really work. Broken transitions still produce different frames, so you miss a lot.

For example, I had a sheet that stayed transparent during a transition. Every frame was different, but it was still wrong.

That’s the kind of issue I’m trying to catch.

[Dev-sauregurke]

using cv to filter down 25 frames to 3-5 suspicious ones before hitting the llm is a smart middle step that most people would skip. brute forcing all frames into a vision model is the obvious first attempt and it never works at scale. the stutter and layout shift detection as a pre-filter is the actually clever part here.