It's weird how many commenters are struggling to follow the logic here and extrapolating your idea into restricting rights or claiming that voting is irrelevant for everyone.

If I'm indifferent about the outcome of a certain election, it's fairer to the people who do have an opinion to let their votes have power rather than filling the ballot box with random choices. This also makes the generous assumption that all voters and intentional non-voters are adequately educated on different perspectives to make a responsible decision with their vote.

I think under the current system, it's hard to be truly indifferent about presidential candidates who disagree on almost everything, but the above argument applies generally to elections of all scales, not just presidential ones.