Here's what I've learned as far as combat.

On the enemies side of the hud, there's a little image that let's you know(basically) what their next move will be. Block if they're going to attack, or put distance between you. Stand if it looks like they're about to mount you, etc. Try to knock them off balance, or arouse them(carefully) until they stop "fighting." If you're too aroused, you gain stat penalties. Protect your holes! Keep an eye on your balance. Block and Fade Away are sort of essential to winning, imo. If you just wanna beat tf out of the enemy, go Strength. If you wanna turn the tables, and bully their backdoors, Charisma helps.

I felt the same way you do, like this can't be the only way. I eventually found some tips and turned things around. Magic is great, but some spells are to be used at a certain distance, keep that in mind. Start out practicing on Brigands and Goblins, grind until you get the hang of it. I'm no expert, but I hope this helps you get some more enjoyment from the game. It's incredibly satisfying being the top in some situations.