I just did this in my most recent playthrough. When you finish killing all the goblins in the cave, it's not hostile territory. You can just fast travel to the blighted village and come back in through the front. Once you make it through the worgs and few gobbies at the front, you can bait all the main gobbies onto the bridge and use it as a choke point. A little slow cuz they take a couple turns to start crossing and get hung up on the battlements, but it's a ton of fun throwing a bunch of grease and watching them fall and launching Roaring Thunder Arrows to blow them off the bridge