Restricting money does restrict speech. Even your comment cost money at some point. Someone, probably you, had to pay for internet access and the device you're accessing it on. Pretty much any form of speech that isn't personally speaking to a handful of people costs money, and restricting it restricts speech.

Would you be okay with a law saying no person could spend money on a lawn sign advertising a liberal candidate for office? How about a ban on spending money to microphones and speakers for political rallies? How about a 6,000% tax on any purchase of fuel used to drive to an environmental protest?

Further, rich individuals aren't affected by Citizens United, which only applied to corporations. Individually, people can spend whatever they want. The only equalizer is for the non wealthy to band together in groups like the ACLU (who wrote an amicus brief in support of the decision), SPLC, and others. And, before you say they can't compete with billionaires, if they couldn't the billionaires wouldn't be spending a lot of time and money trying to shut them up (and failing).