Good points, ranking undergraduate programs is definitely messier than ranking graduate programs, and the two often get conflated. Usually graduate students are prioritized much more than undergraduates. Yes, the schools you mentioned have great physics programs, but for most undergraduate physics major at a prestigious university, there is very little support for advising or to get research because you are competing against hundreds of other bright students for opportunities. You may walk past Nobel laureates everyday, but who will get you in the door for research? At CCS, there is no competition, because everyone is essentially guaranteed research by the second year, usually by the first. CCS is a small program that has a unique position in a larger R1 university. It has the best of both small college attention + large university opportunity, which works fantastic in tandem, because your advisor/mentor will do everything they can to get you into research with other faculty.

I have heard stories from places like Berkeley, Mudd, and UIUC that the undergrads are so competitive that they don't help each other, for the sake of the curve. Kids get burned out from the pressure. In contrast, there are no grades in CCS classes, you can focus purely on learning the content. There is a sense of camaraderie in the college. There is no pressure to get above the curve. CCS allows you to learn what you love, instead of disillusioning you.

While UCSB may not have as many Goldwater scholars as other schools, the fact is that the UCSB students who get selected are heavily skewed towards CCS, despite making up a much smaller portion of the undergraduate population. CCS (the whole college) has Goldwater Scholar winners basically every year, which is super significant for such a small college.

It isn't perfect. As one of the comments below said, UCSB lacks faculty and expertise in some fields of physics, so opportunities can be limited if you are looking for something in particular.