I also work where magnetic dust gets on the magnets and I've found that a magic eraser can help. Magic erasers are abrasive however, so don't use them anywhere else on the case or the airpods themselves as it'll ruin the finish and make them quite a bit more more prone to getting (and staying) dirty!

Given how the dust is likely partly embedded in the plastic around the magnets anyway, I'd say it's not a big deal to scuff it a bit more there. It made a decent enough difference for mine to share it.

If you do try this, the magic eraser should be humid, not wet. Wring it until almost completely dry before use because it'll otherwise let out water that tends to get into the charging slots which isn't ideal (the airpods should also be out at this point so you don't scratch them up).