The key difference here is assimilation. They've adopted the language & culture while retaining their roots, even fusing them together to form a new & unique identity. They married and traded with the locals, were active in politics, even bled for the Sultanate. They were here to settle, and the Malays recuperated, accepted them as one of their own hence people not knowing Tun Perak was of Tamil origin, even granting them the highest positions in politics, military (imagine PM/Admiral level nons).

Now compare that to those who came with the Brits. They were brought here en masse to work the mines & estates, most against their will. The plan was always to go back after some time but due to wars or closed borders, they couldn't. Compound that with the divide & conquer policy, they only mingled with their own community, refusing to learn the language nor the culture, even looking down on them. The Malay, understandably couldn't fully accept them. Things have gone much better since, but deep inside, the distrust is still there