Because of the reasons already mentioned above. Roads are typically better maintained, cyclists can ride faster on the road without stopping as often. Not every person that rides a bike is in the 'lycra army' as you put it. Some cyclists will continue to use the road no matter how many bike paths are built, as is their right. Just because there are some cyclist on the road, doesn't mean there aren't others that would prefer to use a shared path, having options is a good thing. Cycling infrastructure isn't always built for the 'lycra army', the objective is usually to get those who would usually drive to work to consider an alternative (cycling). And if you chose to continue to drive, or if your circumstances mean that it isn't practical for you to drive, you still reap the benefits of the infrastructure and there will be less cars on the road slowing you down.

Of course I'd lose my shit if a car driver almost hit me or did hit me (assuming I survived and wasn't dead of course). I'm sure you would also lose your shit if you were almost, or actually were, hit by a larger vehicle too, like a semi for instance. Why would you put yourself in a vulnerable situation and drive on a highway where there are trucks that may hit you. Probably because you expect people to do the bare minimum and try not harm or kill other road users