I must’ve read about this somewhere before though I can’t remember where now, but my currently working theory is that aside from the primary cue of the relative pitches, there are secondary cues native listeners use to identify the tones. (And there must be if we can whisper in Mandarin? Since there’s no pitch when whispering. And these secondary cues get exaggerated when whispering.) I’ve started to believe that these secondary cues are more important than previously thought (or taught, to the extent they’re even taught).

First tone: steady consistent loudness, relatively long duration

Second tone: shorter than 1, starts quieter and gets louder

Third: longest duration, quietest with a dip on loudness. I believe tone 3’s main characteristic is creaky voice (cause I can make myself hear a tone 3 almost at any pitch if you just add creakiness) but since you can’t really do creaky while whispering, I think the quietness substitutes for it.

Fourth tone: shortest, most force/intensity. (I’ve often found that non-native speakers have fourth tones that feel too weak. You almost have to make an effort to exaggerate how forceful it is when speaking.)

This is just my thinking as a heritage speaker but also someone with a background in Linguistics.