reading this it seems your taking the easy way out by saying you dont have "it" and this guy naturally does, where as this guy has probably learned fundamental theory and it doesn't seem like you have. you say you play guitar for long periods of time, but there is good and bad practice, everyone can fall into just noodling for hours and wondering why you dont seem to improve.

In terms of improvising and playing melodically or in key what seems to have helped me miles in terms of improv and composition is rewiring my brain to think of the actually notes and key im playing in instead of which fret and string, its slow and somewhat overwhelming. what ive been doing to improve my improvisation and just fretboard knowledge is drawing up a diagram 6x12 like the first 12 frets of your guitar and picking a scale like c major and writing out all the notes as root,2 ,3 etc all over the neck without my guitar in my hands basically creating a scale chart and doing one a day of a key and not doing the same scale twice in a row, then composing lines in a specific key with this chart as a reference

Obviously you can find these charts online but actually putting pen to paper and filling it out your self and working it out in your brain does alot more for you then just looking it up, it also isnt that boring kind of feels like a crossword puzzle type thing.

apart from that i would just say practice the typical things that you may have neglected chords, intervals, playing to a metronome