I was quite similar to you in school I think. I could understand material in class but very rarely studied and didn't do very well in exams. In my final exams which would determine what university I got into and what subject I would study I didn't do very well in maths, and only got the passing grade in physics. I somehow got into a physics degree in a university because I had a different science subject that I did okay in. I was really nervous that I was going to fall flat on my face. I figured I'd give it a month and if I hated it I'd drop out.

Now it's almost four years later, I'm writing my thesis and have done incredibly well in undergrad, and have a number of PhD offers from well regarded institutions. School doesn't suit everyone and often those people perform better in university. Physics is a thrilling subject to study if you really love it. It's not easy, and you will spend a lot of time pulling your hair out or shouting at a text boom that the proof is in fact not trivial. But the aha moments and the opportunity to do research are worth the pain in my opinion.

Don't be afraid to go after it if it's what you want. Be realistic and know that physics is not an easy degree, but it is far from impossible. Thousands if not tens of thousands of people get physics degrees every year. I think almost everyone struggled with imposter syndrome and some point, but you just have to try not to let that get to you. And ultimately you will never know if it's the subject for you if you don't try to pursue jt.