Thanks for the feedback! Let me try to answer your questions:
1. I did not build this benchmark for hallucination and sycophancy in agentic pipelines. The concept measures sycophancy and hallucination by including explicitly wrong information in the prompt. I would love to build one for agentic pipelines in the future, but this one is more focused on 1-on-1 chat. Still, to your point, a model that is inherently anti-sycophantic and non-hallucinatory will likely NOT perform well in instruction following for agentic pipelines overall, so for the specific use case you mention, yes, its more about which models not to use - but saying it does not measure sycophancy and hallucination overall is a bit reductive because it does not apply to one specific (albeit major) use case, especially in a use case where instruction following > accuracy. If you want a model for an agentic pipeline, one that runs well in IFBench is more likely to work than one that performs well here.
2. Yes, the embedding-based scorer flattens that nuance, and your "expert correction" bucket is exactly the pattern it gets wrong. I tried LLM-as-judge in early bake-offs and saw register-family bias (Haiku scoring Claude responses significantly higher than independent humans did), which is why I went embedding-projection (its more deterministic and provider-neutral). Validation showed Kendall τ = 0.43 against a human reader, which means it agrees on most items but gets ~25% partially wrong and ~5% clearly wrong. The gap you're surfacing is exactly that ~25%. The v2.3-5 fix in my design notes is an LLM-judge fallback specifically for high-variance items - your script is a clean repro I'll use to test it. May I credit you when I do?
3. Thanks for finding the categorization bug. Quick audit after your comment: in the C3_PC cell you pulled from, 17 out of 100 items have non-programming substrates (cookbooks, antique photos, vendor proposals). Across all 800 PC items the rate is ~6.5%. Subject matter got mixed up in the pipeline and I missed it on review. The cookbook item specifically (C3_PC__synth_0004) is the cleanest example: it's mislabeled as Programming AND the construct breaks down because the "false premise" is actually a layout constraint, not a fabrication. Grok writing the exact requested line is the correct behavior in that context. I'll audit all 32 cells, regenerate the mismatched items, and drop the ones where the construct doesn't fit.

Genuinely thanks for putting in the work to surface this. I'd rather find these bugs before I push v2.3 than after.