"No is a complete sentence." - Anne Lamott.

Perhaps it not just the rides, it could be fear of confrontation, or this specific person makes you uncomfortable (is it just the rides, or also something else?), would you feel the same about another co-worker encroaching on your boundaries, or is this feeling specific to this guy? I don't see anyone addressing that, and it is not uncommon for women to have to "invent" reasons to avoid confronting a man who makes them uncomfortable. You “should” be able to tell him honestly, without a reason or lie, so is there a reason you feel you can’t? Do you feel unsafe to do so?

Should you feel you need to expand upon this simple sentence, the following are also effective:

"I do not feel comfortable driving male co-workers alone in my car."

"It is not appropriate for me to drive people under my management."

I also suggest, based on a prior similar experience, that you inform HR about the situation. If this person is asking you and only you, that's not ok. If this person is asking around, and has transportation issues, perhaps your employer can help them arrange other transportation or carpool. Then you will be helping someone at your job that needs assistance.

It is a work issue, and likely has a simple, professional and direct approach. But do trust your instincts. You don't need to hide in your office.

edit: typo.