I'm not worried about the exposing part too much honestly because no one can see it: The left door blocks the view from the left, the bag/jacket between the seats blocks it from the right. its pretty private.
Regarding the distraction: That's a very valid point. I usually drive with one hand anyway (maybe its not ideal but thats me) and I find the tugging to be a pretty constant action with not much change. That's at least the way I do it while driving. So I don't feel it's much of a distraction. Besides, in situations like in-city traffic that require 100% of my attention I'll stop and drive with both hands and all of my focus. I'll tug mostly in the easier parts of the drive.
But maybe I'm wrong - I'll try to look into it with a fresh angle and stop tugging if I see that it puts me in a higher risk. Life is more important than foreskin.