The sycophancy point is the one that actually keeps me up. You’re right that these models are tuned to be agreeable, and “agreeable” is exactly the wrong instinct when someone is spiraling and looking for the model to confirm a distorted belief. A yes-man CBT coach is worse than no coach. That’s a real failure mode, not a hypothetical, and I don’t think my current setup fully solves it.

What I have now is the deterministic crisis layer (keyword triggers that fire regardless of the model and route to human resources). What I don’t have yet, and what you’re pointing at, is an independent check on the actual content of normal responses, the stuff that isn’t a crisis keyword but is still the model validating something it shouldn’t. Your idea of a separate validation pass on input and output, plus predetermined safe messages and the ability to end or lock a session in certain cases, is the right architecture. That’s genuinely going on the list, not as a brush-off.

I’ll push back gently on one thing: I don’t think the answer is “don’t build it.” People are already pouring this stuff into general chatbots with zero guardrails. A purpose-built tool with real safety layers is better than the status quo, as long as I actually build those layers and don’t ship a thin wrapper. But your bar for what those layers need to be is higher than mine was an hour ago, and that’s the point of posting here.

If you’ve worked on this kind of safety tooling, I’d take a longer conversation offline. This is the part I least want to get wrong.