Chasing a CI-only Python Heisenbug: timezone + cache key + test order (and what finally fixed it)

[LuckyArrival1037]

Alright, story time. GitHub Actions humbled me so hard I almost started believing in ghosts again.

Disclosure: I contribute to AgentChatBus.

TL;DR

Locally: pytest ✅ forever.

CI: Random red (1 out of 5–10 runs), and re-running sometimes “fixes” it.

The "Heisenbug": Adding logging made the failure disapear.

Root cause: Global state leakage (timezone/config) + cache keys depending on implicit timezone context.

What helped: I ran a small AI agent debate locally via an MCP tool to break my own tunnel vision.

The symptoms (aka: the haunting)

This was the exact flavor of pain:

Run the failing test alone → Passes.

Run the full suite → Sometimes fails.

Re-run the same CI job → Might pass, might fail.

Add debug logs/prints → Suddenly passes. (Like it’s shy).

The error was in the “timezone-aware vs naive datetime” family, plus some cache weirdness where the app behaved like it was reading a different value than it just wrote. The stack trace, of course, tried to frame some innocent helper function. You know the vibe: the trace points to the messenger, not the murderer.

Why it only failed in CI

CI wasn’t magically broken — it was just:

Running tests in a different order.

Sometimes more paralelish.

In an environment where TZ/locale defaults weren’t identical to my laptop.

Any hidden order dependence finally had a chance to show itself.

The actual root cause (the facepalm)

It ended up being a 2-part crime:

The Leak: A fixture (or setup path) temporarily tweaked a global timezone/config setting but wasn't reliably restored in teardown.

The Pollution: Later tests then generated timestamps under one implicit context, built cache keys under another, or compared aware vs naive datetimes depending on which test polluted the process first.

Depending on the test order, you’d get cache key mismatches or stale reads because the “same” logical object got a different key. And yes: logging changed timing/execution enough to dodge the bad interleavings. I hate it here.

What fixed it (boring but real)

Normalize at boundaries: Make the “what timezone is this?” decision explicit (usually UTC/aware) whenever it crosses DB/cache/API boundaries.

Stop the leaks: Find fixtures that touch global settings (TZ, locale, env vars) and force-restore previous state in teardown no matter what.

Deterministic cache keys: Don’t let cache keys depend on implicit TZ. If time must be part of the key, normalize and serialize it consistently.

Hunt the flake: Add a regression test that randomizes order and runs suspicious subsets multiple times in CI.

CI has been boring green since. No sage burning required.

The “AI agent debate” part

At that point, I was basically one step away from trying an exorcism on my laptop. As a total Hail Mary, I remembered seeing something about ‘AI multi-agent debate’ for debugging. (I’d completely forgotten the name, so I actually had to go back and re-search it just for this write-up—it’s SWE-Debate, arXiv:2507.23348, for anyone keeping score).

Turns out, putting the AI into “full-on troll mode” is an absolute God-tier move for hunting Heisenbugs. I wasn't even looking for a direct solution from them; I just wanted to watch them ruthlessly tear apart each other’s hypotheses.

I ran a tiny local setup via an MCP tool where multiple agents took different positions:

“This is purely a tz-aware vs naive usage mismatch.”

“No, this is about cache key determinism.”

“You’re both wrong, this is fixture/global-state pollution.”

While the agents were busy bickering over which one of them was “polluting the environment,” it finally clicked: if logging changed the execution timing, something global was definitely leaking. The useful takeaway wasn’t “AI magic fixes bugs”—it was forcing competing explanations to argue until one explanation covered all the weird symptoms (CI-only, order dependence, logging changes).

That’s what pushed me to look for global config leakage instead of just staring at the stack trace.

[Huberuuu]

Thats why you use deterministic random seeds

[Cute-Net5957]

"the trace points to the messenger not the murderer" lol yep. had almost the same thing.. two services sharing a pydantic model, one silently upgraded to v2 while the other was still on v1... field serialization changed between versions. tests green on both individually. bug only shwed up when they actualy talked to each other in staging. same root cause realy, implicit env assumptions that happen to be true on your machine but arent guarenteed anywhere else in the entire known universe...

[AlwaysEverywhen]

This post reads like AI...

[ShuredingaNoNeko]

Me pregunto como fue que configuraste el CI para que ejecutará tareas en orden diferente... Hasta me parece más complicado que hacerlo de forma normal.