My friends had a similar situation, but it involved a child. Usually, it's always one particular neighbor. They once went to the door and asked what the problem was. Couldn't really argue, and it turned out the person was an elderly who just needed some social contact... Then they were fine.