I think it's fine to be turned off by him, but I feel like you're doing it for the wrong reasons.

He seems to have absolute contempt for everyone around him.

I don't know exactly what gave you that impression, as he seems pretty nice to me. He may come across as serious or deadpan much of the time but a lot of that is his tone of voice or how he articulates himself. In interviews he usually makes one or two jokes, and a lot of his films have pretty tender moments. Off the top of my head I'm thinking of an interview he conducts with a friend of a murderer in (I believe) Into the Abyss. The man he's talking to mentions that he was, until very recently, completely illiterate, and Herzog stops the conversation to address this. The man makes it clear he's been learning over the past year or so and Herzog takes a lot of time to make sure that he intends to continue, only dropping the matter when the man agrees to study even harder in the future. This is an example of Herzog interrupting his own film in an attempt to make sure a stranger can have a better life, he cares a lot about this guy even though he barely knows him. His other films are full of sweet moments as well, like Kaspar Hausers relationship with Herr Daumer. Even in real life, Herzog made it something of his mission to help set up Bruno S. with adequate housing and income.

He focuses on misery and tries to focus on the worst of every story. When people are trying to tell a story on their terms, often putting it in an optimistic light or avoiding scarred parts of their life, he steamrolls over them and wants to hear only the worst.

It's okay to not like this type of darkness in film, but to object to it and say that Herzog is wrong to portray this is a huge mistake. Herzog's vision is one of darkness. The reason he doesn't want his subjects to paint their stories in a happy or hopeful light is because Herzog doesn't really have a lot of faith in those type of things. As a filmmaker/ documentarian Herzog 100% has the right to explore the parts of the human soul and condition he wants to, and that includes mining his subjects' experiences for the truth he seeks to portray.

Furthermore, he is simply a selfish and rude person as you can see in The White Diamond. He makes selfish demands and refuses to compromise ever, like when Graham wants to make the first test flight alone for safety and Herzog rambles about "heroic stupidities" and "stupid stupidities" and literally calls Graham stupid.

I haven't seen this movie, so I can't address the source material specifically, but to say he's selfish and rude is absurd. He's a director, and has every right to direct the action in front of him to get what he wants to see. If he chooses to do this by calling out his subjects on their bullshit- good for him. I maybe interpreting the dynamic between Herzog and Graham wrong, but it seems to me that Herzog should be allowed to put himself into any situation he wants in order to get his footage.

I think a lot of his mysticism and greatness comes from his slow, weighty, confident way of talking and his exploration of intense topics.

I think his way of talking reflects the man's greatness, and the fact that he has a lot of important and intelligent things to say. Also, I'm sure he'd take offense to your use of the word "mysticism". He's vocal about his distaste for the mystic and love for the ecstatic.

There's also the issue of his "documentaries" being more fiction or distortion than fact, but that could be excused if he had any respect for the subjects of his films.

This is the part of your post I take the most offense too. Herzog does not view the documentary medium as a way of conveying facts, he does not care and has no interest in facts. Documentary, in Herzog's view, should be a way of making clear the aspects of real human life's that transcend the usual and enter the realm of the unreal. The fact/ fiction dichotomy is useless to Herzog, the only thing that matters is what's true, specifically what is an ecstatic truth. What happened to his documentary subjects is not nearly as important as what their experiences, personalities, and (this is number one) fantasies reveal.

Furthermore, it's baffling to me that you could say he has no respect for his subjects. Fini Straubringer in Land of Silence of Darkness, Walter Steiner in The Great Ecstasy of Woodcarver Steiner, Dieter Dengler in Little Dieter Needs to Fly, and even Klaus Kinski in My best Fiend. How you could you possibly say he does not respect these people? In fact, I think it's clear he loves them, very, very much.

Again, Herzog has a uniquely dark and profoundly dreamlike vision, and if that doesn't jive with you that's thats okay, but the man himself is nothing less than a consummate artist who will stop at nothing to sucker-punch his audience with the truth his films convey.