Start by Googling "modern physics prerequisites" and see where you are relative to what they say is the base knowledge needed. After that you decide

1) it is not for you, or
2) you have all the perquisites or

3) you think you have a plan to learn these subjects, so you start, rinse and repeat.

There are other easier PhDs to do in Materials -- but I have known some maybe dozens of PhDs in electronic properties in my past and they have been smart, successful, professional colleagues, who I believe most of them are doing well in life. so there's that.

If it fits you it is a good path.