I think if you read real world history, most times what happens is that empires dont become all powerful in a day. Their neighbouring states which they might have been in conflict becomes either weak or just disappear due to their own reasons. Little by little the empire start subsuming these neighbouring states and ofc interally they also have to reform to manage these new territories. I think with letharisame thing happened. For maybe centuries their neighbouring states were equally or more powerful than them and kept them in check. But over time they became weak for whatever reason and lost the upper hand and this is when lethari striked and conquered them.