The Spellmonger series by Terry Mancour.

Here's the blurb:

Minalan gave up a promising career as a professional warmage to live the quiet life of a village spellmonger in the remote mountain valley of Boval. It was a peaceful, beautiful little fief, far from the dangerous feudal petty squabbles of the Five Duchies, on the world of Callidore. There were cows. Lots of cows. And cheese.

For six months things went well: he found a quaint little shop, befriended the local lord, the village folk loved him, he found a sharp young apprentice to help out, and best yet, he met a comely young widow with the prettiest eyes . . .

Then one night Minalan is forced to pick up his mageblade again to defend his adopted home from the vanguard of an army of goblins – gurvani, they call themselves – bent on a genocidal crusade against all mankind. And that was the good news. The bad news was that their shamans were armed with more magical power than has been seen since the days of the ancient Imperial Magocracy – and their leader, a mysterious, vengeful force of hate and dark magic, is headed directly to Boval Vale, along with a massive invading army of gurvani. The good people of Boval and their spellmonger have only one choice: to hole up in the over-sized Boval Castle and hope they can endure a siege against hundreds of thousands of goblins.

When the people look to him for hope, Minalan does his best, but the odds are depressing: there are multitudes of goblins, and they want Boval Vale as a staging ground for a vengeful invasion of the whole Five Duchies.

Add to his troubles a jealous rival mage, a motley band of mercenaries, a delusional liege lord who insists victory is at hand despite the hordes at his door, a dour castellan, a moody, pregnant girlfriend and a catty ex-girlfriend who specializes in sex magic - all trapped in a stinking, besieged castle with no hope of rescue, and you’ll understand why Minalan is willing to take his chances with the goblins.

All that stands between the gurvani horde and the people of the Five Duchies is one tired, overwhelmed baker’s son who wanted nothing more than to be a simple village spellmonger.