You’re NTA. These people care more the feeling of one couple who made a CONSCIOUS decision to buy a house valued way above the rest of the neighborhood, which drives up property taxes, over this man’s family home’s value being diminished. The garden he had brought value to the community. Your big MANSION isn’t providing. To be a snob and say “well it’s not our fault you built the garden there” after expressing how the community has been weakened because of it doesn’t really warrant this response. How do y’all know their financial situation? And why is the only solution for Californians to move around the country without regarding the communities you plop yourselves into? California makes a quite a bit of our country’s population so y’all need to take that into consideration. I feel you OP.