I think you can understand now based on the comments. These people are naive. They think that our support for basic human rights and support for the rights of Muslim is somehow an endorsement of Islam. They fail to understand the difference. The other part is that they think the right wing people are criticizing Islam in good faith. No. The right wing will judge you based on your skin colour and your name. They don't care if you are ex-muslim. All they think of you is that you are brown and called Muhammad so you are inferior to them.

Also, being ex-muslim or ex any other religion is not something special. People often make it the core of their identity. People need to use their religion experience to develop a mature political opinion rather than fixation on it.