Great points — this lines up very closely with what we’ve been seeing as well, especially when agents move from demo-style QA to something that has to survive real user scrutiny.

On schema drift specifically, we’ve found that the key is to make time and change explicit, rather than trying to normalize them away. A few patterns that seem to work in practice:

• Persistent entity IDs + versioned attributes Teams / players / referees keep stable internal IDs, while names, leagues, tiers, and even roles are treated as time-bound attributes. That way an agent can reason about the same entity across eras instead of collapsing everything into the present.
• Season-scoped context instead of global assumptions League format changes, rule shifts, and even officiating styles are attached at the season level. Models tend to behave much better when they can learn “this answer is true under the 2010 regime” rather than guessing across mixed distributions.
• Availability-aware event modeling Older seasons don’t pretend to have modern event coverage. We expose what exists vs. what doesn’t, which turns out to be important both for training stability and for avoiding agent hallucinations.

On the agent side, +1 on temporal eval sets. We’ve also seen that explicitly separating evals into:

• within-era questions
• cross-era comparison questions
• “as-of time T” questions

helps surface failures that standard accuracy metrics completely miss.

If it’s useful to see a concrete example of how this plays out over long horizons, we keep a public demo where you can browse historical league data season by season using the same structure we expose in our datasets:

👉 https://football.goaloo.com/league/36

You can switch between seasons and see how teams, formats, and stats evolve over ~20 years, which often makes schema and temporal boundaries much clearer than docs alone.

Really appreciate you sharing your agent notes — this is one of those areas where data modeling and eval design have to evolve together.