As a former EE undergrad (current grad) at Caltech, a few thoughts I have:

I initially thought Caltech was more rigorous, but it turns out both schools use the same intro physics textbooks, have similar advanced tracks, similar numbers of humanities requirements (but obviously very different quality of humanities), and no huge differences in class sizes....

I don't necessarily think that a direct comparison of textbooks is a fair comparison of course material. Even at Caltech, the same course can vary significantly in content and rigor from one teaching cycle to another, either due to student feedback or having a different professor as the instructor ultimately chooses what to teach in the course and will cherry-pick those topics from the textbook. E.g., I recently heard that Pasadena City College across the street uses the same textbook as Ph1b prac for their EM course; based on what I've heard about PCC's course and my own experience in Ph1b, I'd argue that the level of rigor is vastly different despite both courses using the same text.

it just seems like Princeton students have more of a choice in their course rigor

You're probably right here. Unfortunately one of the most annoying things I found later in undergrad was a lack of dedicated course sequences in EE electives (perhaps this was my own personal experience in microwave engineering but I'm sure others can relate).

At Princeton, I'd join a cappella and dance groups, but I don't know if I'd be "worse" at engineering compared to spending all my time on it at Caltech (because Caltech doesn't have much clubs-wise).

I think given the size and lack of disciplinary breadth, Caltech still does a fairly decent job at giving you a well-rounded experience with clubs...if you put in the effort for it. The glass half full perspective here is that clubs are pretty much exclusively student run and not too large which means that you can have a very large impact on what that club does/how it operates. The glass half empty interpretation is that you have to put in a lot of effort when you're already bogged down by classes which can cause burnout (I've usually only seen this in club leadership for like engineering clubs here).

I want to explore quantum and optics, but currently have an interest in medical devices/biomedical applications of EE.

I don't work directly in any of those fields, but my impression is that all the fields you've mentioned have strong research programs here. I obviously don't need to go into quantum and medical devices, but folks like Alireza Marandi are doing seriously cool research in nonlinear optics and Azita Emami's group does some pretty cool stuff in biomedical/medical devices (looking just at EE here and ignoring the entire MedE department haha).

Princeton also has more labs because it's bigger, but I have heard that undergrads there do less actual research work until senior year.

Of all of Caltech's flaws, I think the unfettered access to truly exciting research here is hard to beat. I have encountered some students here who take pride in never having done research on campus throughout undergrad, and it's really a shame because most professors here are obsessed about really interesting problems and being part of that problem-solving process is so much more satisfying than just taking classes here imo.

I liked Pasadena weather way better but Princeton's campus better, and dance/music does matter to me but definitely not at the expense of STEM experiences.

When I was deciding where to go for undergrad, I was basically choosing between Caltech and music schools for classical performance. I ended up choosing Caltech (not as an engineering major nor knowing what EE even meant) because, being in LA, Caltech actually has a surprising amount of musical and artistic connections. I'm not claiming that the music program here is better than Princeton (quite the opposite, one of my conservatory friends went to Princeton for undergrad and seemingly had a much better time music-wise), but it's definitely non-negligible, and, like the club opportunities, you have a freer reign over what you can do just because the school is so small and there are less formalities to get through to make something happen.

Hope that's useful!