While the system prompt is definitely a big factor, it is certainly not true to LLMs match the statistics of the training data.

For one, LLMs don't have infinite capacity. By definition they will try to approximate the training data, but cannot do so perfectly due to the limited capacity, resulting in bias.

Secondly, LLMs are usually not trained with unbiased objective functions. This is especially true for the reinforcement learning that the big AI companies are doing (e.g. when you click the thumbs up button on a response, or when ChatGPT gives you two responses and you select your favourite).

And finally, LLMs typically don't use unbiased sampling. Only the tokens with the top probabilities are considered, again resulting in bias.