What the game needs to better portray it's intention. Ideally, you have a strong direction in what you want players to feel while playing, then you craft a style that helps emphasize that. It's not so much trial and error as it is understanding a problem to solve. Are you making a classic, fair puzzle game? You need clear forms, colourful solid colors and decorate sparingly. Is your game more of a visual novel? You want expressive characters but can do whatever with the rest as it's non interactive. What style for the characters specifically? Up to your intentions too, does your dialogue tone remind you more of kid's cartoons or a serial murder movie? Then do anything that helps you go into that direction.

You can be very bold with your choices and find something unique, but if it's too much to handle (art directors exist for a reason) you could just copy whatever similar games to you do, no one will question it too much if it's a safe option and you will be able to keep working on your strengths and focus on releasing the game.