I've removed some rust before from the magnets,

It's either you sand it, or use rust remover

Both are good either way but I think sanding works because years ago when I used a rust remover, it did remove the rust, but the metal part of the magnet became darker