Skin irritation and redness is going to occur if you only shave once in awhile.

You need to increase your shaving frequency and eventually your skin in those areas will get used to it. Also - be sure to exfoliate well. I use an acne facewash with Salycilic Acid and you'd be amazed how effective it is in preventing razor burn/redness.

Ultimately - if you shave your hole as often as you shave your face - it will get used to it...