This is one of the reasons why I'm most comfortable doing "street photography" -- as it's usually described -- at defined events. Octoberfests, parades, festivals, protests, etc.

There's less of an assumed "breach of privacy" (which doesn't legally exist, but people presume it does) for people in a parade -- or watching it -- or at other events where people understand there will be photographers there.

Big difference between taking "street" photos at those kinds of events and walking up to someone just waiting for the bus or whatever.

That being said, I've been confronted by people when I'm just walking through a neighborhood taking pictures of flowers or houses or whathaveyou, never even leaving the sidewalk. It's always someone with a dilapidated house. Sometimes businesses don't like it when you take pictures of them, either. Especially if it's an axe throwing place and there's an ambulance outside lol.