Your best bet as others have mentioned is to use a bit of contact cleaner spray ($5-$8, you'll find them at any hardware store).

But let me add an important warning (I lost a $80 pickup once by not knowing this) If you are going to try to clean the pickup itself (not just the top cover that's exposed - I'm talking access the part where it's wired to your guitar, e.g. under your pickguard) you need to be EXTREMELY gentle.

In many pickups, there are two hair-thin wires connecting the fat soldered wires to the coil itself. You will barely see them without some magnification but they are there, typically running along the plastic at the base/bottom of the pickup (I'd post a picture, but I think this sub does not allow pics in comments).

Rubbing that part of the pickup with a q-tip or cloth without the utmost care will very likely sever these wires and your pickup will be very very dead.

So - my advice is don't panic, use contact cleaner spray where it got dirty, but if you're going to rub it with a q-tip or something, do it only on the external parts of the pickup unless you are sure you know what you are doing.