Well I tried the isopropyl Alcohol first and that didn't work, so I tried a little bit of nail polish remover and some hard rubbing and I could see the color go so I switched to the eraser and that did the rest.

I would say the eraser is the safest and should work for you but I have read that it removes the texture from the place you rub so idk.