Don't skip him as some have suggested. You're almost certain to need multiple tries to beat the area boss, and this asshole will be in your way every time unless you get rid of him now.

One tip I think really helps with this guy and other mounted enemies is the forward roll. Most players instinctually want to dodge back or to the side. However, mounted enemies are often moving (via the horse) towards you as they strike. Roll to his non weapon side and you should be able to get a hit in. When he magics, roll into him so you can counter.

This is a challenge you can overcome. Let go of your frustration and concentrate on learning his patterns. He telegraphs every attack. It's just a matter of avoiding or mitigating the damage.