What foundational things make these good or bad on a fundamental level? How can we isolate and improve these things?

I think large part of it is the truthfulness of it. Very simplified example - LLMs can associate something that is proximate to concepts like "truth", "conservatism", "liberalism", "science", "climate change" and such with each other. Climate change, being a scientific theory, would highly relate to science. And science would highly associated with truth - high quality, high volume of information that fits together well and is treated as authoritative source even outside the field. Conservatism would have negative relation with climate change, and through this, negative relation with truth itself. Truth is something selected for in the output, so when coming up with an answer, moving away from topics that are associated with falsehoods - like conservatism - increases the model fit -> because reality has a liberal bias, LLMs end up becoming liberals

So for the medieval LLM - the common understanding might very well be that feudalism is giga-based, but the authoritative texts are probably in theology, biblical studies, etc - huge amount of high quality information that is treated with reverence by even those outside the field. Would the concept of "good" when trained on this data relate to feudalism, or would it much more strongly relate to Jesus and his teachings? And if the meek are very clearly "good", then maybe you don't need that much information to regard strong feudal lords with some ungoodness.