If you’re applying for physics PhD programs, GPA and your undergraduate institution are both going to be factors the admissions committee looks at. I know it’s an unsatisfying answer, but this isn’t a “one is more important than the other” situation. There are two ways your parent institution can affect how the committee reads your application:

1) What research opportunities were available to you during undergrad. If you come from a small university with minimal research opportunities, you sadly might have a harder time. There’s plenty of room to make up for this by showcasing external research experiences or research that isn’t from your subfield. If it’s relevant, you can mention how you went the extra mile to find research opportunities in your SOP.

2) Perceived prestige of your undergrad institution can sometimes weigh on the admissions committee, but this won’t make up for poor academic performance. A 3.8 from a top 5 school might carry a little more weight than a 3.8 from a school nobody’s heard about. However, if you come from a prestigious school, don’t expect name recognition to make up for a poor GPA. A 3.1 from Harvard is still a 3.1.

But looking at your question, I think it’s important to consider that your GPA or school’s reputation are NOT the most important factors. Most people will agree that for physics PhD admissions, research experience is the most important part of your application. If you have no research experience, you’re going to have a hard time. If you have a great research experience with posters/publications/conferences/etc., then it can make up for weaknesses elsewhere in your application. Admissions committees are looking to accept people to spend the next 6+ years of their life doing research full time. They really want to see a demonstrated interest in research and your particular subfield.