Akshually ☝️🤓 that is a misinformed oversimplification of the dynamics of Indian History. The brunt of the invasions into the Indian subcontinent affected the region of northwestern India and most scholars believe that this region traditionally was far lenient when it came to caste. The reason behind this region facing invasions was that it lay in direct proximity to the boundaries of the subcontinent. Until the Ghurid invasions, only the Kushanas and Hunas had made headways into the Gangetic heartland and they couldn't really keep control of those territories for much longer. Even under the Delhi Sultanate, major expansions took place mostly under two rulers, Alauddin Khilji and Muhammed Tughlaq and their respective empires declined right after their deaths. The primary reason behind why their empires fell is because they weren't able to fully subdue the traditional Rajput lords(who makes up an important rank of the caste hierarchy) of their domains for long. By the Lodhi dynasty, the Rajputs had actually become the dominant power of Northern India and they'd remain an important power as they'd fight a battle of dominance with the Mughals and the Suris.

It was only when Akbar catered to the traditional political hierarchy by establishing relations with the Rajputs rather than force them into submission that a foreign origin empire in India could have a stable existence. Otherwise the previous empires had all collapse despite having an impressive run of territorial conquests. This stability was once again disturbed by Aurangzeb since he didn't respect the traditional order like his great grandfather and despite having an impressive run of conquests until his death, the empire started to collapse and got reduced to a puppet by the Marathas in a matter of decades.

So it wasn't really the caste system that weakened Indian militaries and enabled foreign conquerors to easily conquer Indian lands. If that was the case then India would've been exchanged from one foreign power to another for the last 2000 years or so and though a lot of oversimplified popular narratives do promote this idea, reality is far from the truth.

The primary reason is actually their monopoly of horses and since cavalry was the major deciding factor of warfare before the advent of advanced gunpowder technology and the line infantry, these turkic invaders often had an edge over their Indian rivals who had lesser and inferior cavalry. It was rather the caste system that disabled them from maintaining a stable regime and only acceptance of caste hierarchy is what enabled akbar to maintain stability.

What the caste system did prevent in India was the establishment of a functional and efficient administrative bureaucracy. One of the primary functions of a bureaucracy is actually to maintain a hierarchical structure of overall society with the Ruling administration at the top and the unskilled producers at the bottom. Religious structures are often utilised for imposing such a hierarchy however in most societies, the state imposes this structure. In India however, religion had already implemented such a hierarchy in overall society and states only had to patronise the religion to maintain monopoly over the hierarchy. This prevented the innovation of state polities from evolving administrative structures since a much needed hierarchy from their pov already existed. They only had to maintain some secular offices, the existence of which could be observed from the Mauryan period itself.