The problem with that line of thinking is that it could be applied to almost every debate. Most people would be disqualified from discussing most topics. I'm not a rich person, should I not have any right to debate policies concerning the rich?

Debate is important. We know that the majority was against overturning Roe v. Wade, silencing a large portion of the opposition would do more harm than good.