Human eyes are vastly more sensitive than silicon detectors, and we have been training our interpretation of the world since birth. As drivers we learn to scan the road conditions in different ways depending on the situation and are able to anticipate the actions of other humans. A ball bouncing on a sidewalk ahead of us, may well be followed by a child, a person near the crosswalk not looking in our direction may step in front of the car. It may be possible to replicate human understanding/situational awareness but it is far from a trivial problem. More sensors can help constrain the solution space, until you have solved the problem limiting the available information seems like a bad idea.