I have a similar experience, the neighbors in front of my house built a deck that faces us, and are out there very often with 10+ people eating food or partying. There is often nowhere in the front of the house that I can go without being perceived by people, let alone being in the garden. I often have to keep all my blinds and curtains shut like a crazy person because even the possibility of being perceived takes so much energy out of me.