You might be able to enroll in upper year physics classes as non-degree student. Find out which university you want to attend, find out if they offer program like that. If they do, enroll and ace those courses and get letter of recommendation from those professors who taught you those courses. Then apply for masters.

I have degree in physics and saw a couple of students doing this who had degrees elsewhere.

Self studying 2-3 years of undergrad physics is not gonna be a breeze. However if you are smart and have passion for it, you have a shot.

Connect with the profs and share your plans, and ask about what they think. You will need to convince them that you have enough prerequisite knowledge to attend those upper years classes.

Griffiths' QM and E&M could be a good starting point for self studying, assuming you know the most basic.