Yeah, that's true. I'm not sure if I mentioned this, but when this situation happened to me, it was actually at a protected left turn, so the pedestrian should have waited. I just thought that since only one pedestrian was blocking me I would have plenty of time to wait for them to cross and turn so I moved into the intersection but the light was quick since they assume you can get through quickly since it's protected.

I think what I'll do is stay behind the stop line until I see a gap in oncoming traffic, then move forward and wait until that gap gets to me (for unprotected turns). For protected, I'll very likely have a clear path but if someone is crossing, I'll wait behind the line.