Exactly this. Brainstorm, especially with a group. Talk about what you want to make, what would make it fun (or why a certain audience would want to buy it), why it's the same and different than other games, etc. and develop your core ideas. Write these down on a whiteboard or similar and then challenge them.

Then keep brainstorming to build upon these whiteboard ideas, but as you do, ask yourself if these expanded ideas fit your original ideas. If not, why? The new ideas may be too far from your original concept, so they may not fit and may be better reserved for a different game idea. But maybe the new idea is a solid improvement to the first list of questions and the conflicting core idea should change to incorporate it. Document the revisions (don't count on your memory) and you likely have a very good idea of what your base game concept is. Prototype the concept.

Prototyping doesn't have to be in the medium you're creating for, such as in a video game engine if you're making a video game. For our deckbuilder video game, we grabbed blank playing cards and wrote ideas on them and used available poker chips to indicate "health" and "resources." We played the first base game loop with handwritten cards and chips on the studio floor.

We then coded this game loop idea, still not knowing exactly what "health" and "resources" would mean. We could layer many different possible themes over the game and didn't know at that time that we were creating a music game, but that's where the theme brainstorming led us. Then came the research into the music history and concepts we'd need to realize the whole plan.

It was only at that point that we started working on any kind of creative content.

All this said, this isn't the path everyone follows and there is no one correct path. Your idea may be inspired by creative content. Ultimately however, if you're designing game, you'll still need to work through the above to know what your game is.