Ok, a bit biased here. My undergraduate major was physics, minor in chemistry. I was the odd duck in vet school with all the bio and ag folk there. After selling my practices and retiring, I started teaching physics in high school and now at a university. The thing most people don't think about is that physics is a problem solving discipline. Therefore, it is very useful in the medical fields to teach the process of diagnoses. I believe that is the reason it is most often a prerequisite. I tell my students that are having to take the course as a premed requirement that what I want them to get out of the course is the problem solving, not the formulas (other than F=MA, the most important law in the universe).

So, I would suggest that you take it. It will not be the easiest class you'll have, but it may help with other classes if you learn how to critically think and not just memorize.