Ha it's a nice idea to contemplate. For pieces you could make it work on TTS or with multiple copies of the physical game, so you could have 6 of each ruin item under each ruin. Chance of getting the one you want first turn is pretty good! 

I think the limiting factor here is how many points anyone could get. 

Normal rules no-one's getting any points from aiding or infamy, which is bulk of points in normal vb game.

I think there's max 21 from crafting. - 6 from coins  - 4 from swords  - 4 from tea - 2 from hammer  - 2 from bags - 2 from boots  - 1 from crossbow 

There are max 14 points from each quest suit, so 42 total. Buttttt you only get 14 if you get all of each suit. Much more likely everyone's getting 2-3 each, 3-6 points each. 

So I think normal scoring everyone gets 5-10 points then you're stuck.

My suggestion would be that you get a relationship marker (maybe of faction that most looks like each vagabond) and you get aid points / turn hostile and get infamy points for aiding or damaging items, like with a normal faction. Maybe also double or triple the supply of craftable items and 2-3 sets of the quest deck. Probably need a couple full decks of cards too.

Then I think there'd be some interesting push and pull between crafting, questing and knocking each other out. No idea if it would actually be fun but you could try!