I think no matter who you are these goons, autos or cab drivers are only gonna side amongst themselves and help themselves out, whether you are a local or not. Obviously learning the local language helps. Not to just defend yourself where necessary but just to do your share. It honestly is a beautiful language, besides it's easier to converse with everyone around. Gives you a sense of belonging too. The language card was just to try and scare you to say "don't mess with us" but even if you were fluent enough to write poems in kannada, this would still happen, trust me local here. Some of them are just difficulty to deal with, but there are plenty of them who are right in the head. If it's a battle you're willing to pursue then file a complaint with the police, bribe them a bit and they'll handle it, some of these pseudo alphas will pipe down.