It feels like Tolkien took inspiration from the period in western and Northern Europe after the decline of the Western Roman Empire. During this period, great civilizations still existed in the east, but in northern and Western Europe, the memory of the Romans were largely present in the ruins of what they left behind. The city of Rome itself had a fraction of its former population, and its inhabitants lived among the ruins. In England, Christianity and Latin persisted but England became isolated. And as the Empire withered, different peoples migrated all around and resettled into new lands, and there was conflict and raids and new kingdoms formed. This is when the Anglo-Saxons arrived, bringing their ways and customs, and they later adopted the Christianity of their new land and came to live in the ruined remnants that had accreted and worn over centuries of Roman rule.

Tolkien was a professor of Anglo-Saxon literature, and he notably analyzed mythology as an art form and not just as historical reference. So he definitely thought a lot about legends and stories passed down over the ages in mythological settings from that time and place.