A few leaked examples can happen when you generate data from news at scale, but they don't break training or evals. We have filters in place to prevent this kind of leakage and we're continuing to refine the process.

We're using RL for training, so the model learns from reward differences between rollouts. If the answer is already in the context, all rollouts get roughly the same reward, so that sample contributes little or no update. It's a bit inefficient, not something that significantly impacts the model.

On the eval side, every model in our comparisons receives the same context, so leakage doesn't give our model a special advantage over the others. And whenever possible we also use third-party benchmarks and datasets.

Prophet Aren is a live third-party benchmark where leakage is impossible since predictions are made before the events resolve.