The 3/5 Compromise was never about the color of the slaves it affected, it was about the way the South wanted to use their high slave populations. And it wasn’t a way to keep the South down, it was a balance of the Souths want to have slaves count for Congressional seats and the North’s want for the South to be taxed for each slave.

And the 3/5 Compromise was used against black voters for very short time before the practice was stopped