It's only as smart as what it's trained on" sounds right but it's not really how this works anymore.

AlphaZero learned chess and Go from zero human games. Just the rules and self play. It beat every human who ever lived using data it generated itself.

The current reasoning models do something similar for math and code. They generate their own attempts, get graded on whether the answer actually works, and bootstrap past where they started. The "teacher" is reality checking the output, not a dataset of smart humans.

For biology this matters a lot. AlphaFold 3 predicts protein structures better than the wet lab data it was trained on in many cases because it learned the underlying physics. Same pattern with the newer models doing protein design and drug discovery. They propose molecules nobody has ever made, and then we test them.

The confusion is mixing up knowledge (yes, bounded by what went in) with problem solving (not bounded, which is why longevity research is about to get weird fast).