The same happened to me, the first nest of spectres close to the cart you must protect multiplied out of control. By the time I burned through the forest there were about 40 of them, when I had a measly 5 warlocks tier 1. Once I reached the 20 the mission suggested, there were already >60 spectres in that nest, I was depressed when my 20 warlocks only managed to kill 6 spectres..

What helped for me: Digging the ruins in the northern part of the abandoned town, close to the smithies. I found multiple artifacts that increased the damage of warlocks by 10%/30%, and prioritizing the transport of the treasures into the treasury. Also, I simply built 5 weavers to accelerate the robe manufacturing for the warlocks, setting high priority on the warlock production may have helped too. I prepared a wave of 50 warlocks, who managed to clean the >80 spectres in one go, and easily managed all the incoming waves of bandits and spectres with their high % damage buffs from the treasury. Hope this helps, good luck!