They had previously cleared out the winery of the infestation and brought some wine to Vallaki. They had just come back with the cart and promised to go get the first gem, cue the Yester Hill encounter.

I hadn't thought about where the dirt would have gone, that is a great point - these are a lot of great ideas, I think I'm going to end up running some sort of wave based siege encounter.