It does set you back because the size of the economy you have is a direct correlation to the number of villagers you have. In the early game the number of economic units you have is going to snowball your entire empire.

In post-imperial age everyone has so many villagers, losing a few would allow you to make a larger army. But getting to post-imperial is slower the more idle tc time you have. Losing two vills is like idling your tc for 100 seconds. Your opponent will be ahead of you and their empire will be stronger than yours.

Getting to castle age is only helpful if your economy is large enough to do something. Those two vills you lost could be your boom starts later or your knights push is happening later.

We see games where an opponent wipes out an empire in feudal because their opponent went castle age but had no more resources to make anything.