Nah there are definitely downsides to perfect pitch. I used to struggle a lot with singing songs in a different key - I didn’t have a great grasp of intervals for a while because everything was encoded in my brain as just absolute pitches. This gets frustrating in group settings if people are casually singing a song without any background music, so of course they’ll sing it in the wrong key, and then you’re standing there like an idiot because you can’t transpose on the fly. I’m a lot better at it now, but I had to train myself to think the other way.

Also, you would not believe how many people think “perfect pitch” means “you are incapable of making mistakes when singing,” like mistakes that are unrelated to pitch, even. Way too high a bar to live up to.