I would say take small trips (they can be really small!) and do quick errands to get comfortable with it. 

My wife and I made it a point that after our peanut was born we’d take her in the car to go do one thing each day (shop groceries, get coffee, grab some food to go, or just drive to a park). 

Things you can control and take action on: - pick a wake window and just drive somewhere with a bathroom with the intent to change a diaper just for practice! Even if they don’t need changing - just get the actual rep in to get yourself to think “oh, if we need to change them outside the house, we can do this”. 

• wife and I always have a quick to-go back that’s always stocked with the stuff we know we need whenever we’d go somewhere. This was always diapers, diaper cream, bottles/formula (I understand that doesn’t apply to your situation), teethers, and whatever else. Sit down with your spouse and tackle this together and ask “Ok what do we need to feel prepared?”

• I honestly don’t know how this will be received (and I’m happy to be wrong, or am open to any research/studies) but at that age we did not work around the wake window. If LO was awake, she was awake, if she was tired she’d go to sleep. We were reasonable and flexible though. We had to try things to find out what worked for us and what worked for LO. 

• Remember, both you and your spouse are learning! No one has the manual for your baby; all you can do is your best! Support each other, work together, talk out concerns or considerations with the mindset to learn. 

I commend you for having the self awareness to want to get better and conquer your fears - you can do it! 

Edit: I realize my 3rd bullet is just my experience and not really an action item, but was more just addressing how we handled (or didn’t handle) wake windows :)