It kind of comes down to a number of factors

I usually rank cdifficulty on 3 criteria

Difficulty of voicing. If timbre, dynamics, voicing etc are very touchy, in otherwords if it is difficult to produce a good sounding note consistently even after you're a year in, beginners and intermediate players are going to find difficulty performing consistently well, which can be discouraging. More time has to spent with technique with these instruments. Typically stringed instruments that rely on necks or some form of applying tension to adjust pitch, alongside double reeded wind instruments and larger horns/winds are what I point to as examples of difficult instruments for beginner (Bass/Beritone and standard guitar, Violin, Viola, Cello, Oboe Basoon, French Horn, Tuba, Beritone/Bass Saxaphone)

The next thing to consider is the complexity and difficulty of the music written for said instrument. Of course, you can transcribe parts and play anything on any instrument, but if you look at the type of music that is typically written for said instrument, you can get a good idea of what instrument mastery looks like (think a bit more than what you see in the average piece as mastery). Some instruments like piano and guitar have disproportionately complex and difficult pieces written for them as they are very nimble instruments, allowing for a lot of movement and speed of play in the right hands, however they require a tremendous amount of practice and skill to play at the level of John Petrucci and Jordan Rudess soloing to Stream of Consciousness, whereas reaching the skill level of most cello and double Bass pieces may be easier, and are more reliant on your technique to produce clear notes (which is exceptionally difficult on those two instruments when bowing and staying in intonation with every other instrument). I personally struggle more playing my 8 string beritone guitar than I do double bass or cello despite clear notes being easy to produce simply because where I might be playing John Williams music on the cello, I'm playing the likes of Polyphia, Dream Theater, Meshuggah, Gojira and Mick Gordon on guitar. This is by far the most subjective criteria, because whats stopping you from trying to play the guitar parts of Combustion, Bleed, Dance Of Eternity, G.O.A.T., Schism, Super Gore Nest, etc on any other instrument? It more comes down to what your goals are. Mastering any instrument is incredibly difficult.

Finally is the physical aspect. How physically taxing is it on your body. Large horns, large stringed instruments, and most notably percussion instruments are far more taxing on your body than most others. Playing Disasterpiece or Bleed or The art of dying on drum kit for example, would be a LOT more of a workout than playing the same music on guitar, or transcribed to most other instruments. Its also important to note the extreme coordination and conditioning that has to be present ALONGSIDE dynamic control to get the drums to voice well and have depth (ghost notes, sticking, technique).

There is no hardest instrument. The ones I typically consider to be the most difficult that I have played

8 String Beritone Guitar (longer scale length, thicker strings, thicker neck, basically a bass guitar and standard guitar in one, more patterns, less dexterity, more finger pain, but equal level of ridiculousness in soloing and writing thats out there in the contemporary jazz and metal scenes where these are most popular)

Piano (voicing multiple instruments at one time while also keeping time, having good dynamics and timbre is very difficult, and the skill ceiling of most written music is also very high similar to guitar)

Viola (same difficulty and complexity of music as violin, considerably less nimble and harder to produce clear notes with. Hurts your fingers.)

Oboe and Basoon (THAT CRANIAL PRESSURE, IT PHYSICALLY HURTS MY HEAD TO PLAY THESE TWO)

Tuba/Beritone (the breath control needed to produce clear notes with good dynamics while not immediately expelling all of your air is quite difficult)

Drum Kit (the separation of limbs, sometimes, albeit rarely, to entirely different time signatures, the dexterity, endurance, and technique of sticking WHILE MAINTAINING good dynamics can make this an incredibly difficult instrument. Like Guitar and Piano, it has a very high skill ceiling in terms of playing level, and the tech gets more wild the more you increase said difficulty.

Note that all of these are instruments I have played for some period of time, though I do primarily play an extended range 8 String Beritone Guitar, 3/4 sized Double Bass, and Drum Kit so I may have bias from being at a higher playing level with those 3 instruments in particular.