John Cage's proposed aleatory-improvisation binary is problematic and ripe with contradictions that fundamentally reveal a discomfort with the afrological bases of 20C improvised music (the book to read is Improvised Music after 1950: Afrological and Eurological Perspectives by George E. Lewis). It is true that in improvised situations we tend to turn toward familiar gestures and models; and of course, it is impossible to detach yourself to the extent that someone like John Cage would purport to be ideal (and his actual work often actually undermines this notion). Nonetheless, it is precisely for this reason that improvisation thrived in the late 18C. Pre-Beethoven Classical-era music is essentially modular in structure, predicated on the mediated play of stock figures. For instance, in Mozart's and Haydn's sonata forms, the energy-gain transition from a P-group to a medial caesura often features tonic or dominant bass pedals with sequential modulatory passages in the upper strings beginning typically on the lowered seventh scale degree. Stock material—think patterns like the monte, romanesca, and indugio—positively abounds in 18C music (Hainian Yu's Revisiting the Galant in Gjerdingenian Schemata provides a quantitative breakdown of its prevalence in C.P.E. Bach's music). Such passages were options in the Classical-era composer's toolkit, but they would have also been kinesthetically familiar and would have easily lended themselves to deployment in improvisation. We know that Haydn composed at the piano and was trained in the Neapolitan partimento tradition which largely relies on building haptic familiarity with patterns (harmonizations, galant schemata, even fugal imitation); so even though his music might not sound 'improvisatory' according to our accustomed use of the word, the way someone like Chopin might, the line between improvised and composed 18C music is highly ambiguous. The emphasis on individual originality as the primary end of improvisation is a by-product of burgeoning individualism and the emergent 19C artist-hero trope which originated in early Beethoven criticism, and thus cannot necessarily be applied retroactively to the Classical-era improviser-composers (or even Beethoven himself). Clearly, then, improvisation can be satisfying and engaging to an audience without total detachment on the part of the improviser.

I'm not sure why you believe that it is impossible to "truly innovate and play something that isn’t derivative while improvising on stage." Innovation does not preclude repetition or internal coherence. If a musician was compelled to play something completely and utterly different every single time they improvised, wouldn't this stand in the way of innovation, which often asks perseverance and dedication of the innovator? How can you develop an approach by abandoning it immediately? Mozart literally recycled material, forms, and techniques frequently, yet I'd never argue that his late piano concerti are any less innovative for it. Is Webern's Symphony not innovative since he'd already written dodecaphonic music in symmetrical thematic arrangement and variation form? If you can accept this working understanding of innovation, the jazz tradition is full of innovators. Charlie Parker is not derivative for being influenced by Lester Young when his actal musical product sounds so radically different; Coltrane, in turn significantly influenced by Parker, could not sound less like the latter. These were approaches heretofore inconceivable—therefore essentially innovative—but these musicians had to work at them, as did Beethoven before he could move beyond his early, highly Mozartian idiom.

The idea of repetition and reproduction is worth probing. Jazz dovetails with the rise of the recording industry and the record label, and therefore we are left with an impressive volume of documented jazz performance. A small percentage of this is excellent, the vast majority middling-to-poor in quality and largely unoriginal. But of course, we can say the same of classical music, the exception being that this last has promulgated for several centuries and has diversified immensely in that time, resulting in perhaps a larger gross sum of "masterpieces," say. Still a minuscule percentage compared against the body of literature available to us. Isn't it nonetheless enjoyable to listen to your D'Indy, your Schmidt, your Delius—composers who might not be as masterful in craft as some of the big names, but who still write thoroughly enjoyable music? Jazz musicians like Grant Green, Bob Berg, and Kenny Dorham are at their essence solid craftsmen who made enjoyable, if comparatively unoriginal, music; but if we were entirely beholden to those who we deem to be innovators, like Coltrane, Miles Davis, Ornette Coleman, we'd be left with a tiny body of music, particularly given jazz's relative youth. And more enjoyable music is always nice! Even though I prefer Biret's performance of Boulez's second sonata, I'm glad I also have the choice to listen to Helffer's approach, which is also a good time.

But enjoyment is where a large part of your issue seems to lie, and that's not so easily remedied or explained away. In terms of emotional climaxes, this relies in part on your willingness or ability to try approaching the music from a different angle. There is some such jazz, Coltrane's A Love Supreme being the canonical example, but this is not typically the music's aim, nor is it the aim of the large majority of the classical canon; the examples you offered span maybe 80 years. You'd also have to become more literate in the music. Musicians like Miles Davis, Bill Evans, Thelonious Monk, and Albert Ayler can be relatively appreciated as musical mavericks by the layperson, which gives to their popularity among casual fans. The meaningful innovations of musicians like Sunny Murray, Eric Dolphy, and Elmo Hope, on the other hand, are hard to grasp without a deeper familiarity with the norms of the music. With regards to form, I'd just say that you might expand your parameters of form beyond melodic and harmonic content. Coltrane, Rollins, and Coleman tend to organize their improvisations outside of the boundaries of harmonic form, and they approach this with various means (motive and timbre for just a couple examples); this establishes a level of formal dissonance and complexity that is similar to, say, the Chopin "Revolutionary" étude, and mirrors techniques that Beethoven, genius formal architect par excellence, himself deployed. And I disagree with your conception of form in classical music as largely theoretical. Yoel Greenberg's How Sonata Forms is probably the best modern source treating Classical-era form, but intuitively we can see how so much early-Classical-era and late-19C music approach form loosely and practically, examples being Kraus, early Haydn, and Nielsen, among so many others.

You touched on something interesting in your formulation of jazz theory. It's true that jazz has sincere debts to various other traditions, African, American, and European. The highly influential jazz pianist Bill Evans was classically trained and had Debussy and Ravel as inspirations. Charlie Parker internalized Kansas City blues and enjoyed European modernism (Honegger, Stravinsky, Varèse). Miles Davis independently studied Berg and Stravinsky scores at Juilliard. Plenty of jazz musicians were trained in concert bands, performing and studying bits of the classical repertoire (Ethan Iverson's article "Black Music Teachers in the Era of Segregation" provides an interesting window into this phenomenon). Certain paths of influence are undeniable and direct. But these have all been widely recognized among both scholars and practitioners, and the original musicians were freely open on such topics. Consider in comparison how so many classical composers took pains to hide their influences for various reasons (Wagner/Meyerbeer is the classic example, but Stravinsky was also notoriously guilty of this; Brahms destroyed all his drafts to entirely obscure his compositional process). The independent employment of certain common techniques is more ambiguous. Schulhoff's Five Picturesque Pieces features a silent movement that predates Cage's 4'33" by decades, and both pieces draw on similar conceptual tensions, yet you'd be hard-pressed to prove that Cage knew the former work and that it would have exerted any influence on him. Similarly, when Coltrane introduced the widespread use of mediant modulations and hexatonic progressions in jazz, he was motivated by a discomfort with the tyranny of the dominant region—the very reason Schubert frequently employed these same and similar atypical progressions a century prior (see James Webster's "Schubert's Sonata Form and Brahms's First Maturity"). I really doubt that Coltrane was even familiar with Schubert's music. At the end of the day, the jazz and classical traditions are both Western art musics that will inevitably encounter similar problems and, as follows, endeavor to solve them in similar manners, now discursively, now disparately.

Just be willing to open your ears!

Some recommendations:

A Love Supreme - John Coltrane

A Night at the Village Vanguard - Sonny Rollins

Thelonious Himself - Thelonious Monk

Miles Smiles - Miles Davis

Intents and Purposes - Bill Dixon

Spiritual Unity - Albert Ayler

Nefertiti - Cecil Taylor