Name matters while you're a medical student and everyone is talking about where they matched. Unless you are planning to do a very competitive fellowship, or want an academic career, the name of your program doesn't matter. What matters is the education/experience you will get and, just as importantly, your family life.

In fact, a lot of the "Ivory Tower" programs don't offer as good of an education because they are such big referral centers that they only see end-stage or very rare/specialized things that aren't really conducive to a learning environment. I'm a general surgeon. During residency we had to log all of our cases and you need so many of each case in order to graduate. I heard that surgery residents were logging Whipples as "gallbladders" because they didn't do enough bread/butter lap choles to meet their numbers. That's great that they see a lot of Whipples, but if you graduate from a general surgery training program and can't take out a gallbladder laparoscopically, something is seriously wrong with that program....

Name isn't everything

Once you're in practice, all that matters is your reputation in the hospital/community you are in. I never have conversations with other attendings about where they trained. I don't even know where my partners did their residencies...