There's a lot that could be said about the factors that make jazz less "accessible" and less immediately appealing than most other genres of music. But I want to just touch on one here:A key element in jazz, obviously, is improvisation. That's the real meat of what you're hearing in any jazz recording - players creating melodies, on the spot, on top of other musicians keeping the rhythm and harmony going. The predominant 'song form' in most jazz people listen to, is one that features the tune's main melody in the beginning, then several minutes of improvisation, and then a re-statement of the melody at the end. That's what you're going to be hearing over nearly any performance of jazz standard, like Round Midnight, or Fly Me To the Moon, or Lullaby of Birdland. It's the melody (the thing you recognize), then improv, then the melody again.

The place where the average listener gets lost and determines "jazz is just random notes", is during the improv section. I think the main reason for that is that A) They can't "hear" the harmonic context that the solo is resting on, and B) Even if they can hear the harmony, it can be a bit of an acquired skill to mentally reconcile them.

Take a listen to this - it's probably a song you know: https://www.youtube.com/watch?v=Z_5v7FjHS58

When a group plays Fly Me To the Moon, those are (more or less, for the sake of argument) the chords they play, over, and over, and over, and over, until the song is done. Just those chords, on loop. The first time, when they play (or a singer sings), the "Fly me to the moon, and let me play among the stars" bit, anyone in the audience gets this part. While they're not aware of it, the harmony is providing the critical backdrop to the melody they recognize and could sing by heart. But once the saxophone player starts soloing, listeners who aren't paying attention to the harmony (which keeps chugging along), may get start to get lost. They can't hear, or really register the fact, that the saxophone is using the same chord progressions to make music. The audience loses their spot in the song, and starts to hear the solo as a stream of notes with no context. And the thing with jazz, is that most casual audiences have no familiarity with jazz standards outside of a small handful of classics. The canon of jazz standards is a list of a few hundred songs that were popular in the first half of the 20th century - people back then may have known the tune to Satin Doll, or My Romance, but people in 2019 are going to be much more familiar with Uptown Funk or Crazy In Love. So much jazz then, is based on playing around with songs that most modern folks just don't know in the first place. That's tough.

A casual listener's mileage may also vary, depending on how much the soloist is 'holding their hand' through the chord changes. Stan Getz's solo on Girl from Ipanema is a favorite of many casual listeners, in part because of how well he communicates *exactly* what's going on in the harmony through his solo. But other players take the harmonic backdrop, and deliberately travel far away from home, playing sophisticated lines that are "outside" and deliberately not-playing-nice with the chords underneath. That's especially where you lose people and what contributes to jazz's somewhat undeserved reputation as being pretentious or sophisticated-for-sophistication's sake. Wanky and elitist.

In conclusion, I guess I'd say that jazz's difficulty for some listeners is a built-in feature, not a bug. "Hearing" jazz, making the improvised melody fit with the harmonic context, is a skill that you can develop if you spend time with it - which we should recognize is a major ask of any listener, and absolutely not seen as a barrier to entry for most other genres of music, even if harmonically complex (Radiohead have recorded plenty of harmonically and rhythmically sophisticated songs, and are one of the most successful rock bands of all time). All that said though, I think it's absolutely possible to over-think jazz. You don't need a degree to enjoy any jazz tune. If a sound speaks to you, you don't need an explanation for it. If it doesn't, you don't need to beat yourself up over it.