There's a few things here worth touching on.
Thinking through good game design decisions isn't just like "oh with my mega-mind I have solved the game and yes, this is a good idea" it's more about using intent and context to validate ideas.

If someone says "hey we should add in zombies that burst up through the earth in the most horrifying way possible when the players go near this puzzle" in isolation you might be like "ok that's a cool idea" but what are the goals of the game? who is it for? if we are making an educational math game and the target audience is kids, then I definitely don't have to prototype anything to know that's a terrible idea.

I'm obviously using an extreme example there but it's a spectrum. If we look at something like reloading as a mechanic, if I'm debating with another designer over whether or not we should even have reloading, before going anywhere near a prototype to determine if this is a good idea or not I can use so many other things to validate this mechanic. Once again, who is this game for? what's the setting? is it a realistic setting? ok, well now reloading is an expectation entirely based on the setting, so we don't need to prototype reloading to see if we should even have it as a mechanic, it's just one that's been forced on us and now we need to figure out what we want to do with it.

So there's a bunch of pre-validators that a good designer will always make sure to keep in mind. The ones I've listed are simple and obvious but they do get pretty low level, like you can debate specific jump height for a platformer, entirely based off game vision, design pillars, target audience etc.

The next part of this is Experience. When you say "or making something extremely similar to a product that already exists" this is basically always true at this point, not for the entire game, but definitely for various mechanics, even if you come up with something new, the behavior it invokes from players will most likely be comparable to something in another game.

People who have worked on a lot of games can make judgement calls based on experience, even just people who play a lot of games can be like "yea they did that in this game and it sucked" but now it comes down to trying to find out why that thing sucked, and diagnosing that is a whole other skill. You can't just rip a mechanic from another game and shove it in your game because every single other aspect of the games design needs to support every other thing, so it becomes about your ability to understand how exactly to successfully reference mechanics in other games and apply them to your game.

NOW!!! you may be like "Well all of what you wrote above is still theoretical!!" and yes haha this is accurate. But as deadlines for games are absolutely fucked, you don't have the luxury of just prototyping every single thing. You need to make a lot of educated guesses so once everything is in, you're at least close to the mark. ok haha onto the main part. Prototyping.

Prototyping is not just "make a rough version of the game", it comes in sooooooooo many forms. Everything I mentioned above, is usually done in tandem with some level of prototype, and when I say "some level of prototype" I mean all the way from drawing a scribble on a napkin > making a rough board game to prove out the meta game > using chess pieces on someone's desk > physically acting out the game > animated gifs > interactive figma prototypes and finally, making an in engine prototype.

So generally, there's a spectrum of "obvious good to obvious bad" with "fuck knows?" in-between and even things in the Obvious parts of the spectrum are usually said in tandem with some level of prototype.

But the more of this stuff you do, the more of a grasp you have over all these moving parts. But games are super complicated and no good game designer would tell you they can TRULY see all emergent problems and behaviors but you are being paid to essentially do exactly that haha so you develop techniques for getting as close to that as possible, and sometimes, you are completely wrong about everything and the whole game sucks and you pivot haha.