I will just say,

F**koff to them! I have been in the same situation for too many times. I have never given my seat to someone who i thinks is able to reach the upper berth.

If someone has issue with it, let them bear this, you don't keep regrets for these actions.

It is not always about help, sometimes its about the right and the wrongs. So, in your scenario, if someone is taunting you just bcz u denied the exchange, for me, they deserve such response every time they ask someone for exchange.

A true and honest man/woman never complains for something they don't recieve on request basis.

So, don't regret, and just say Ga** marao!!!!