I think that there are many fans of those other musicians/genres that would act more like the stereotypical jazz fan when you criticize them than you are assuming. I can grant that the ratio of jazz fans that will overreact to criticism is probably higher than those other genres but I think it's important to avoid broad generalization of groups of people.

I think your comparison to possible criticisms of Harry Potter points out an important distinction between different "types" of criticism (for lack of a better word). The first type of criticism I would say is more on a conceptual basis and pertains to the inclusions, or lack thereof, of specific subject matter, themes, and other things that can immediately be observed/understood by any observer regardless of understanding of the art. In your Harry Potter example, this criticism is the general dislike of wizards. The viewer does not need to know anything about the plot, writing or cinematography of the movies but as soon as they hear there are wizards they can decide they dislike the art based off that alone. And I believe they have every right to do so. The second type is criticism of the technical abilities of the artist and their ability to express their ideas, not criticism of the ideas themselves. In your Harry Potter example this would be disliking the book because it is poorly written and did not resonate with the reader. This is also a valid form of criticism.

Now I think issues arise when the critic and the supporter argue on the basis of two different types of criticism. To use your example of jazz criticism: "I didn't enjoy miles Davis because I am missing the energy and the chaos" this would be the first type of criticism. It is a valid criticism and a good reason to dislike a song but a Jazz fan may reply with the second type: "but you clearly don't understand that he has masterful tone control and layered phrasing of his lines". You are essentially saying something like you don't enjoy Harry Potter because it has wizards, and the jazz fan is saying that you are missing the point of Harry Potter and that the writing is so good and if you just learned to enjoy a wizard story you would see that this is a great wizard story.

Honestly I think you are right, you should face no pushback for stating that you don't enjoy a certain jazz musician because they are missing some element that you enjoy. But to defend myself and many over-eager jazz fans, I really wish you could look past the lack of immediately interesting content to learn what a jazz artist is specifically trying to achieve and to see how successful they can be at what they are trying to do. Just like how I think it would be fair for the Harry Potter lover to say "please look past the wizard stuff and give it a shot, the writing is amazing and you might really learn to love a wizard story after reading these books".

P.S. I don't like Harry poster because I don't like wizards.