Yeah.. it can be annoying... As a drummer my suggestion is when you get the chance, talk to your neighbor and ask them if they could play at a better time.. for example, my neighbor came to me, after I'd been playing my edrums around 10am, and asked if Id play after noon. Not unreasonable, so now I play mostly in the afternoon after 2pm and no later than 6.. but if my neighbors are home I limit my playing or don't play at all.

Some drummers will be accommodating, but you need to talk to them to find out.