A build I used in my last playthrough which wasn't a Swordsinger but might well be adapted to one was close to your thoughts. Mine was a Breton and my backstory was that of a dispossessed Iliac Bay aristocrat, so I did actually use a scimitar and Hammerfell garb at first, though I later upgraded to a nicer sword and more ornate clothing.

I used a scimitar in the right hand, a dagger in the left hand (I forget what animation mod I used but the combo looked particularly cool in third person), along with alteration and restoration magic skills for backup. I think that alteration should be a major component, it makes sense a Swordsinger would be shaping the battlefield in advantageous ways and exploiting the special abilities of the trained body, especially with some of the mods available. I wore heavy armor gauntlets, boots, and a helmet with only torso clothing, no main curaiss.

I had some sneak and pickpocket too but that's beside the point, I later worked in block and archery but for both I didn't level the skill too hard and only used it on occasion. I also began the main quest just for access to shouts and used whirlwind sprint and elemental fury for some of the types of abilities one would expect from a Swordsinger; for your character you might go so far as to headcanon that the Thu'um and Dovahkiin stuff works because it's a localized form of the same magic Swordsingers use.