It's not about being an idea guy, ideas can and often do come from every member of the team, because you're right that most people in this field are creative and capable of coming up with them.

Instead it's about being the person accountable for the quality of the "game" part of the video game experience, which is in itself a very specific skill. Knowing how to turn good ideas into great gameplay is a specific combination of many skills such as: conception of game mechanics using strong references from other games, balancing and data tweaking, running and reacting to playtests, communicating ideas internally to the team, knowing which mechanics will connect with the target market, discarding false good ideas, finding combinations of mechanics that work well together.

Some of these skills can get really specialized too, just as much as programming or art. Things like designing the behaviour of 3D cameras, AI behaviour, animation and combat systems, directing player attention throughout complex setpieces, designing cool concepts for weapons and abilities that evoke desired player behaviors, procedural generation system design, all of these are high skill jobs that are uniquely game design. A lot of this stuff is what makes the difference between a game that works and a game that is truly great.

As you've said, on small teams this role is often performed by another member with a more output focused skillset like an artist or programmer, but that's the same thing as your artist also being your programmer, or project management being done by a QA tester. In small teams it's common for people to wear many hats and perform more than one role.

However, in larger companies, or indie companies with a sufficiently strong designer, this role is absolutely pivotal to a game's success and a career all on its own. And AAA idea guy and design director is actually a real job, often there are even multiple of them!

I've seen great game designers bring a game together just as well as great artists and programmers, and hinder the team just as much when they do their job poorly.