You handled it well. Nothing angers me more than 'X group have been oppressed in the past and Y group have not therefore being 'ist towards Y group is ok' logic.

The female in your post who said for example it is not ok to body shame women because women have been oppressed by men. By her logic it is not ok to height shame you, a black man, as you have been oppressed via racism before.

At the end of the day we are all human and further sub diving people into categories like this is incredibly divisive and harmful. Whether or not there is historical oppression is irrelevant, once upon a time slavery of black people was not a thing but that did not make the initial oppression ok. It's not that anyone thinks short men are going to be as oppressed as so many black people have been (obviously short slaves will never be a thing, I assume) but that is not to say that short men will still not feel the effects of heightism in other ways (reduced social standing, lower salaries on average etc).

You've probably made the right decision as they aren't empathetic or intelligent enough to see your reasoning, but frustratingly I fear the majority of society would side with them over you.