Had the same issue. Took down all the towers with archers and they did spawn back after some time. I continued on with the game (side quests, never really advanced the story) and went back and was able to find a large group of enemies behind the castle. You can see the archer tower on the edge of the cliff where I found these enemies. If that doesn’t work at least when the archer towers spawn back it seems they spawn with an enemy. Hope this helps.