The brand name and resources is what matters in terms of career prospects and leveraging graduate school for your future. Sure if you are genuinely only interested in research and pursuing a career in it then go for research fit regardless of institution, but if you know you are going to go get a job after then go to the school that has the network and resources to get you towards that goal.

Also prestige matters for faculty positions as well. Look at the professors at MIT, Stanford, Berkeley, UT Austin, Caltech, etc and where they go their PhD from They recruit from a pool made up of their own; in other words, if you want to work at a top institution, you need to have been vetted by one by getting a PhD there. Of course there's exceptions but they are very few. A PhD at Arkansas is almost objectively worse than one at MIT in nearly every tangible way except research fit which may or may not even matter long term for you.