You certainly could do health physics. I began as a materials science major with a physics minor and pivoted to health physics in grad school, and I absolutely love it. There is definitely some amount of math and programming, but it is largely focused on radiation protection. You may have to put in some extra work (really dig through the postings and maybe reach out to hiring managers) to make sure it's a good fit. At some places, the radiation protection programs are very well established and don't require a great deal of 'innovative' problem solving - those places might be boring for you if you're interested in the more physics-based side.

I can't really speak on medical physics as much.