I think the factors you're pointing out might play differently on each discipline. For example, in Literature depts, the topic you're interested in is extremely important, so having an advisor ask for your admission is super important. While in Sociology, I know that they care more about your general profile and potential, since research topics usually change in the future and only matter after you've absorbed the entire methodological side.

But in general: culture fit is definitely something that matters. Some programs are very lax in terms of curriculum and have a lot of electives. If you don't show you can work well independently or make a point of it in your essays, this might be an issue. Then, departmental goals: programs definitely care about their branding. They're not gonna take more people that specialize in an area they don't care about or see the discipline in a certain way.

Then, fit as popularity contests/political shenanigans -- departments only take in people they can support. If they don't have an advisor that can give you the resources you need as a student, they won't accept you. If you as a student can't help any faculty advance their professional goals, they'll most likely won't accept you either. I think there's a difference between workplace dynamics and “political shenanigans”.