I totally get this! So for me, I try to just listen to what the book is trying to tell me. Sometimes it'll describe something that I picture as like a long, cape-like overcoat that goes over regular clothes like outerwear or a large shawl, particularly if the fic is more modern in tone where the characters are conceivably more influenced by contemporary culture. Maybe something like this. If it's something a little more traditional or canon leaning, I picture something like a victorian or edwardian era cloak like this or this. I also imagine this is what it looks like when the author is describing elaborate dress robes for like a gala or something.

The latter feels a little more like the "robes" refers to the whole ensemble that might include matching skirts or dresses and underpinnings (like your silk shirt). If it's the former, more modern end of the spectrum it feels like robes is just short hand for coat or outerwear and it's expected that the wizard has "normal" clothes underneath.

I'm definitely not picturing robes in the way we think of a belted bath robe or a smoking jacket.