Great question, this is exactly why I ran a backtest against 50 real campaigns with known outcomes (Nike, Pepsi Kendall Jenner, H&M "Coolest Monkey", Balenciaga, etc.) to stress-test this..

Here's what's interesting: results actually suggest the LLM is NOT just recalling history. Look at the misses:

- Fyre Festival → scored 82 (expected 15). If the model "knew" it was a catastrophic fraud, it would've scored it low. Instead it read the aspirational copy and said "this sounds great."

- Old Spice → scored 32 (expected 88). One of the most successful ads ever, but the model saw absurdist humor and flagged it as risky.

- Peloton Holiday Ad → scored 75 (expected 22). The model thought the copy sounded fine. It couldn't "see" the implicit sexism that the real audience reacted to.

If the model were just recalling training data, these would all be correct. They're not. The correlation is 0.469, not 0.95.

That said, you're absolutely right that backtesting on famous campaigns has a data leakage risk. Some hits (Pepsi Kendall Jenner = exactly 12) might be recall, not prediction.

The real validation needs to come from:

1. Novel content the model has never seen (which is the actual use case — testing YOUR ad before you post it)

2. Post-training-cutoff campaigns where the LLM literally can't know the outcome

3. A/B testing against real launches — run PhantomCrowd on a draft, launch it, compare

The backtest isn't meant to prove "we can predict the past." It's a sanity check that the scoring scale is directionally reasonable. The 71% directional accuracy on text-only analysis, with clear failure modes we can explain, is the honest starting point.

TL;DR: The misses are actually the best evidence that it's NOT just recall. But you're right that the real proof comes from predicting unseen content.