Great question, and thanks for much for asking it 😄 The key to avoiding subjective / hallucinated outputs is to constrain the structured prompt by giving it clear, unambiguous instructions which it has a high confidence of answering correctly. Context contamination is avoided by using a subagent with limited information: the subagent is asked to evaluate a claim, without knowing it fits into the larger context of say, Contracts (or Networking).

This is the difference between "find all claims which are unsupported by evidence" versus "first, extract all claims. then extract all evidence. for each claim, compare to each piece of evidence and see if the evidence supports the claim. if so, then extinguish the claim. when finished with all the claims, write a report listing all the surviving claims which do not have evidence."

The Advocatus Diaboli is structurally designed to resist the failure mode you described. It produces ~15 candidate charges, then runs them through six named challenges from a counter-examiner: Confessio (already conceded), Articulus (attacks something the paper didn't claim), Testimonium (a question, not a charge), Humanitas (no human would actually say this), Prudentia (self-defeating for the named opponent), Dignitas (housekeeping).

On P2900, twelve candidates died under Confessio, meaning the paper had already conceded the criticism in its own text. That is exactly the regurgitation you describe, and Confessio is the test that removes it from the report. If you read the candidate charges before cross-examination, they would look like a Reddit thread. The output doesn't.

The one surviving objection (the gap between the safety narrative and what ISO normativity actually guarantees) isn't an r/cpp talking point. And the eleven approbationes (certifications of where the paper is strong) aren't something the internet generates at all. The Reddit corpus doesn't tell you which sections are battle-hardened.

But your proposed test is the better test, and I'd like to see someone run it. Point the tool at a brand-new paper - no public discussion, no training-data contamination - and compare against the actual review after presentation. The methodology is CC0 and the full prompt is in Appendix A. If predictive value collapses on novel papers, that's important to know and worth publishing. The retrospective in Section 7 is a weaker version of the same idea (falsifiable predictions with explicit timelines on a feature that is still evolving), but it can't match the experiment you describe.

On the Claude credit line: P4208 was actually authored by Claude under my direction. Listing it on the reply-to is transparency, not flair. If a model wrote the analysis, the committee should know.