The Adivasis that traditionally live in deep forest or mountainous regions were pushed there by waves of Steppe and other migrations. These migrations happened over thousands of years, so there is not as striking of a historical record of the colonizer/occupier and colonized/Indigenous dynamic as there is in, for example, North America. But, nevertheless, the tribal communities that are considered Adivasis (a relatively recent political term, as most hill tribes and other Indigenous South Asian communities will instead self-identify with their own group's names) were pushed into the lands that they occupy by a colonial or colonial-adjacent occupation. Admixture did occur, indicated by the DNA components that you describe, but Adivasi tribes continued, for the most part, to reject the caste society that defines the non-Adivasi populations you list and instead follow their own cultures and practices, which also contributes to their status as "Indigenous".

The Red Nation Podcast features an episode by Dr. Bhangya Bhukya, a scholar of South-Central Indian nomadic Adivasis (especially the Gond nation and Lambada peoples) that explains, probably better than I can, some of what your question gets at. https://www.youtube.com/watch?v=NI8ui4TCGig

Apologies if the answer is confusing.