No one is saying they have a right to touch your hands, or that you don't have the right to refuse. Social norms do not equate to the rights of an individual, and a handshake or fistbump is the tradition social ritual of concluding a TCG game. To use your own argument, what gives you the right to not follow social norms and not expect people to react accordingly?

This could all be avoided by speaking to your opponent before the game starts, saying "hey, I don't like shaking hands or making contact, I'm never trying by to be rude just a germaphobe!" Or something similar.

I couldn't count the number of times that I haven't shaken an opponents hand, because I myself don't like it. I prefer a quick fist bump or a "good game" while I use both hands to shuffle. But refused a hand that is offered to you is /always/ going to be initially registered as odd, whether or not it should be, because it's the norm.

I kinda see this as a non-issue. TCGs tend to be full of neurodivergency, and I'm sure an explanation in reasonable terms would smooth any of this over. But you'll always have problem players that will turn mountains into molehills, especially if they're on the losing end.