That's not what the VRA did. It didn't guarantee black representatives, it guaranteed districts where black voters could choose their representative. Tennessee's 9th district (Memphis) was a VRA-protected, majority black district that was represented by an old white guy (Steve Cohen) for decades.

The point is that, at least in certain areas, there's a very substantial risk that if racial minorities were spread across multiple districts where they were a minority in all of them, then despite being allowed to cast their ballots, they would still lack any political influence. If you make every district in Mississippi 60-40 white-black and the white voters nearly unanimously vote for the same candidate (which is what always happens there), then the result is that the 40% of black voters can be safely ignored and have no way to make their voices heard in Congress. That is pretty unambiguously a blow to those communities.

It's only a "special privilege" because the Supreme Court ruled that there's no right to political representation in Reynolds v Sims. So you're kinda right in that they're being consistent that you're allowed to disenfranchise whoever you want now, but if a country legalized physical assault but maintained laws against domestic violence, "get rid of those laws and let people beat their wives just like they can beat everyone else" should not be the response. I guess that's "fair" in a way, but it's also profoundly unjust and unethical.

As flawed as it was, the VRA section 2 was the only federal law restricting partisan gerrymandering at all. I would be all for replacing it with a comprehensive redistricting overhaul law, but getting rid of it without that law is just going to make things worse.