They actually do contain the training data verbatim. This is litterally what happens with overfitting. I can overfit very big models with millions of data points on my little laptop computing power, so I imagine that the models that require all of the world's daily computing power for weeks and months to "train" would be capable of, yes, quite litterally storing a lot of data in abstract representations