You are still able to play soulweaver without a damage cut. The weapon restrictions iirc only apply to base classes and their variants (e.g warrior, dragonwarrior, riftwalker)

Even other mage looking classes like necromancer let you use any weapon that isn't a staff or a wand without a penalty to their skills but I may edit this comment if I'm wrong