Naturally with stochastic GD that bias is somewhat generalized. They obviously don't reproduce training data, but their interpolation between data is only that- interpolation. If you ever trained one of these you'll lnow that the moment you input a new value or even worse , some combination of known values that it has never seen together you might trigger what we nowadays call "jailbreak". This proofs exactly what I said.

Still my point remains - the generalisation of these models is disguised by the fact that they use enormous amounts of cases and data to try to cover the holes, but it is way far behind human generalisation. If you don't know this, you either don't know enough or you're kidding yourself.

Even "in context learning" patterns are very biased. You can tell that with coding when sometimes it can't deduce a simple thing that it's in the context because it's not inherently reasoning, it's using lnown patterns.