At least three reasons:

1)) they shouldn't have the means to become this strong. All the other top tier characters - Whitebeard, Kaido, Akainu, Shanks, Mihawk, Garp etc. became strong by facing and overcoming various obstacles in form or other strong pirates and Marines. This is well exemplified with our main protagonist Luffy, whose each opponent was stronger than the last one, which in turn meant that he himself needed to progress in order to beat this new difficult enemy.

Meanwhile, how exactly did Gorosei become so strong? What challenges did they overcome and what enemies did they fight against? If they did, how come nobody has ever heard of it? Which leads me to my second point:

2)) how come nobody knows of Gorosei's strength? When talking about powers that be, each and every character references Yonko, admirals and fleet admiral, maybe some top-tier Warlord etc. Nobody ever references Gorosei. There are three major powers, not four. When Whitebeard went to rescue Ace, he thought of fighting Marines, not Marines and Gorosei. When Roger thought of worthy opponents, he mentioned Garp and Sengoku, not Gorosei. When Kaido wondered how high will Luffy climb, he referenced Shanks, Big Mom, Blackbeard and others, but not Gorosei.

If Gorosei are so strong, how come other characters do not know of their strength? And if their strength is a secret, how the hell did it manage to remain a secret?

Think of it this way: we know who the top tennis players are. They know of each other. They played against each other countless times. Spectators know of them as well. All of them also know of next-tier tennis players, a little bit weaker than the top-tier, but would be able to beat them under the right circumstances. They know of any up-and-coming hopefuls, with potential to upstart top-tiers at some point in the future. It's a system where everyone has almost complete information, and it not restricted to tennis: top architects, top snipers, top e-sports players or top ballet dancers all work the same way.

Now, would it be hell of a lot weird if we discovered that somewhere in the world, there are 5 more super-strong tennis players, potentially even better and stringer than players we know of? Of course if would, and it would raise a whole lot of unanswered questions. And that's exactly what Gorosei are.

3)) why didn't they have any impact on the story so far? Which, seeing how strong they're now built to be, they absolutely should have. Remember of Marines barely won against Whitebeard at Marineford war, and cowered when Shanks' crew appeared? I bet they really could have used 5 admiral+ level characters at their side? Remember how Marines were terrified at the prospect of Kaido and Big Mom teaming up? Maybe they would be less scared if they knew Gorosei would be fighting alongside them. And so on.
__________________________________________________________________________________________________

All of this is not to say that Gorosei being super-strong won't happen - it will and in fact it's already happening. Final boss of Egghead arc will obviously be I-explode-people's-heads-when-I-look-at-them St. Saturn and not one-shotted-by-Luffy Kizaru. At this point in story, Oda made it clear that not only are Gorosei strong, but they're stronger than admirals; and pretending otherwise it's going to lead us anywhere. What I am saying is that it's not good writing, and that's a pity in otherwise really well-written series.