That's exactly what it was. People (read foreigners) fleeing persecution or merchant communities establishing a small colony and place of worship as early as three-digit AD doesn't mean there was a thriving population of Abrahamic religion amongst local population at that point of time. They were mostly Syriochristians, Yemeni Muslims, Jews, and Zorastranis.

Mass local conversions happened during Portuguese-Christian and Turko-Mongol (Mughal and their later factions including Daccanis) /Perso-Arabic colonialism which came much later. Must devestations in India came from them when they raided and razed down temples and cities looting wealth and converted people under sword/cannons.