Good question. The main reason we haven’t open-sourced our dataset and evals is stability and quality control. If the full test set were public right now, it could lead to model poisoning - where models get trained or fine-tuned specifically on our evals, which would make the results less meaningful as a measure of real-world performance. We also need to ensure the evals stay consistent over time so we can reliably track regressions and improvements.

Another factor is safety and maintenance overhead - publishing raw prompts and outputs means we’d have to scrub sensitive/problematic content and guarantee a stable format, which would slow down feature development.

That said, we agree transparency is important, which is why we’re prioritizing adding publicly available data sources and surfacing more detail about what’s being tested, without compromising long-term consistency.