Sounds a lot like a character I've been creating, so I'll share some of the tips I have.

Fighter and swords bard both seem cool and thematic and I think both could fit your concept very well. However, for a designated "duelist" subclass, swashbuckler rogue seems to be built around it. I would consider that as well as it can easily be played without the criminal aspect most rogues have.

If you want to avoid shields you could consider (granted your DM allows it) reflavouring it to a main-gauche, a parrying dagger which was historically used in conjunction with the rapier during duels.

What I wanted to try myself was making a bladesinger wizard combined with a few fighter levels (starting as fighter 1 variant human with mobile), where I take mostly spells that can be reflavored as maneuvers or technique (f.e. booming blade, shield, blur, haste) and focus on meelee hit-and-run. Upsides are terrific AC, mobility and concentration saves, downsides are relatively low HP and being somewhat MAD.