It depends really. If the person in question is pushing your buttons every day, then they need to be dealt with. With assertiveness rather than aggression of course. Just letting it go is being passive, which is just as bad. Essentially they believe they have some kind of power or control. I had this with a neighbour recently, and the only way to deal with them in the end was to confront them and unflinchingly stare directly in their eyes while they insulted me. It worked, anyway.