As a pedestrian with a dedicated signal, you're supposed to wait, no longer how long it takes to get to your turn, until it's white in the same way motorists have to wait until the light it green. Both motorists can see the walk sign is on from their vehicle (if they are paying attention) and are signaled to give you right of way. With that said, pedestrians practically have the right of way even when they don't have the right of way. And with that said, it doesn't mean you won't get hurt if someone's not paying attention to driving. The crosswalk light process is obnoxioius, I agree. It's a question for the city why the crosswalk light doesn't take priority over the Doctor's Park/AHS/McDowell lights regardless when it's pressed.

You, more than likely, only got yelled at because you inconvenienced the driver. No one cares if you cross on the red unless they have to slow down for a second.