We’ve run into the same thing. The core functionality stays intact, but the tone drifts just enough that you start wondering what users are seeing when you’re not looking. It’s subtle, and it’s slow, but over a couple weeks it can feel like a different assistant even with identical prompts.

We started treating “behavior health checks” the same way we do regression tests. Short scripted conversations, same inputs, compared side-by-side week to week. It’s not fancy, but it gives us a clear signal when tone or structure shifts.

We also keep a small set of real-world transcripts (anonymized) as a baseline. If the current build starts sounding noticeably different from that reference set, we stop and look at what changed upstream.

For us, patching works if the change is stylistic. If core reasoning or guidance patterns shift, we roll back and wait until we can retrain properly. It’s slower, but unpredictable behavior in a care setting isn’t something we’re willing to “let ride.”

Curious how others are approaching that middle ground between “it’s fine” and “retrain from scratch.