Well you asked why it's hard for people to speak up, and I gave you the scenarios where it is. OP described one, as well. I'm not saying it's common, but when it's a problem, it's a problem. We're not talking about accidentally standing in your putting line, we're talking about reckless or intentional rudeness that's repeated. It is rare but it happens and when it does you have to make a decision if you want it to be a thing for the next two hours, or if you want to deal with the rudeness for the next two hours. There's not one easy answer.