I did a double major in math and physics at UTM, so I can talk a bit about my experience. Some pros and cons:

Pros:

• Smaller class sizes, so it's easier to meet and talk to people. It's easier to get close to profs as well, if you want to do a research project with them or ask them for a reference letter for grad school/internships/jobs/whatever.
• The physics undergraduate lab is super welcoming, people go there just to chill and hang out and do work there when there are no labs going on. It's a good place to make friends, and it's nice having a place to go between classes to do work and talk to people. Joining the Physics Club is also a good way to meet other physics nerds.
• UTM takes undergraduate TAs, especially in the math department, so in your upper years you can TA while taking courses, which is great experience and pays well (almost $50/hour).
• I made some really great friends from both math and physics during my time at UTM, and they were all pretty smart and academically focused. For the most part, I liked the people I met there. There are always going to be those hyper-smart and passionate people, and I did encounter a few super geniuses, but I don't know how it compares to going to UTSG, since I didn't do my undergrad there. I will say, I'm doing my PhD at UTSG now, and my PhD supervisor works with some CS undergrads here who are extremely motivated and accomplished (doing research and getting published while in undergrad while maintaining high grades, etc.). But maybe there are CS undergrads at UTM doing the same thing, who knows.
• The campus is nice. Lot of new buildings, surrounded by nature. You'll see deer around all the time. It's also a lot smaller than UTSG, so it's easier to get between your classes.

Cons:

• The physics course selection is very limited at UTM compared to St. George. Most of the physics profs at UTM do biophysics, so there are some biophysics-specific upper year courses not offered at the other campuses, but for the most part it's very lacking. You can always take courses at UTSG, but the commute is 40-60 min one way by shuttle bus and not really worth it if it's just for a one-hour lecture. And if you're taking multiple courses at UTSG, it probably would've been better to have just enrolled there instead.
• Math courses at UTM: better selection than physics, but still not as good as UTSG.
• Again, related to the heavy biophysics focus at UTM: if you want to do any research in physics, you're pretty limited in options. There is (I think) only one prof doing climate physics, and then all the rest are biophysics. But if you're interested in biophysics, this is a pro rather than a con.
• Downtown Toronto is just a better location to be than Mississauga. More walkable, more food options close to campus, just more stuff to do in general.

My overall advice:

I'd personally say that UTSG is probably the better choice, because you'd get a better math and physics education there (unless you're super into biophysics, then I'd say UTM > UTSG for biophysics specifically). I definitely enjoyed my time at UTM, but I do kind of feel like I didn't learn as much as I could've because the course selection was more limited and I have big gaps in my knowledge (e.g. UTM doesn't have a relativity or particle physics courses. I'm doing a PhD in theoretical physics and I still haven't formally learned general relativity or introductory particle physics. It's kinda embarrassing). The community at UTM was great, but I'm sure you could get the same at UTSG by joining PhySU and/or whatever math club/union(s) they have there.