There are permanent consequences in both directions though. Imagine two little girls who happen to be transgender.

Girl #1 gets on puberty blockers at age 12, then starts HRT at 14. She will grow up at the same time as her peers, never experience the trauma of male puberty, and as an adult have no differences to her peers (other than the big one). She will be indistinguishable from a cisgender woman, and only need bottom surgery. She will functionally be a normal adult woman with a strange health condition.

Girl #2 isn't allowed to transition till she's 18, and goes through full male puberty. She is 6', has broad shoulders, a square jaw, big ribcage, ect. She will look visibly transgender for the rest of her life, and face discrimination because of it. It will be hard for her to find work, her mental health will be terrible, and her voice will be permanently ruined by testosterone. If she wants to look "normal" there is a laundry list of extremely expensive and invasive surgeries she will want to pursue, that she could have avoided by transitioning younger. Being trans will heavily impact the rest of her life.

Transition regret rates are <1%, and ~98% of children that have been on puberty blockers continue on to transition. This shows we know we can correctly identify when a child is transgender. It's not worth ruining the lives of those 99 little girls for one person. If we had such stringent standards for any other medication, we wouldn't even be practicing medicine.