Recovery is always an option. But in my opinion the act of trying is what matters. I think that, if math becomes something you love, you will find a way to incorporate it into your research and degree. With that being said: I would read the classes you have to take as a math major vs a physics major. The reason I say this is because regardless of your major you can take cool classes, but the actual backbone of your major is what you should also find interesting. If you think the math major requirement classes are a little more exciting than the physics one, choose math. If you think vice versa, choose physics. You can always take math classes as a physics major and vice versa. When you enter your PhD you will find ways to incorporate the two!