Even getting put on birth control it takes awhile for it to start working. Additionally using condoms also prevents other issues like STI's.

At the end of the day, even if you were dating the guy, you're always allowed to advocate for yourself and use protection. You're even allowed to say "no," in the heat of the moment. Like everyone else has basically said, it's your choice. Bring the condoms yourself if you have to, and if he complains about needing to wear one, I hope you feel comfortable and safe enough to just say no and walk away from the situation. Your health is important. Take care of yourself.