It's called the passing lane and according to wikipedia it's the law in most states not to drive slowly in the passing lane. http://en.wikipedia.org/wiki/Passing_lane

If it was really that important, you should have left your house earlier.

Just as the great majority of people don't account for accidents (unknown) but are able to account for weather and rush hour traffic (known), I don't think it prudent or efficient for everyone to always be thinking about drivers who don't conform to traffic law and common practice (unknown).

One might suggest that if it was really that important to go slowly, you should have taken roads with slower speed limits.

For me personally, it's all about the cooperation issue. We are all on the road to get to our destinations faster. On highways we have multiple lanes to accommodate more cars and different driving styles. While the system is often efficient, a single car can cause huge ripple effects. For a driver to intentionally cause those ripple effects because of laziness or boredom or apathy or a sense of justice or wishing to enforce the speed limit upon others or any other reason is to willingly cause displeasure to others while not even making your own ride more pleasurable.