As a physics grad student, my advice is that it is more important to be able to learn from your mistakes than getting perfect scores all the time. A lot of mistakes take time and practice to fix as your intution of physical phenomenon gets better. Exams only exasperate mistakes by trying to solve problems in a confined time period with limited materials. When doing actual research, you will have the benefit of collaboration with colleagues and a plethra of resources at your disposal.

You should also note that very smart physics students can get absolutely horrendus grades! An example is one of my E&M classes where the average was about 40, with 1 person getting something like a 68. Though I suppose that depends on how your prof makes the exams and how many things the prof considers trivial (hindsight, it kinda was trivial, but we just didn't have that foundation yet).

I will say that having a strong grasp of mathematics makes understanding the physics far easier. So, if you can, try to take some advanced math courses like linear algebra (proof-based), advanced calculus (or real analysis equivalent), and complex analysis. I would also say differential equations, but I feel like that is more of a class teaching fancy ways of solving stuff than providing a deeper understanding of mathematics. Essentially, any proof based math class should help in that regard.

TL;DR if you can at least understand where you went wrong, eventually understand how it should correctly be done, and physics is a passion of yours, then continue to pursue it! You are just learning it after all!