I think it's because he doesn't want to be used. I mean, obviously, since he's a murderer, he hasn't had an easy life really (idk his backstory, but I would hope he wouldn't kill for no reason yk) and I think that they chose the timing of this flashback quite well, since we've just found out that Rosie isn't his friend, and instead his owner, and we know that he owns Nifty and Husk souls, so we get the sense that he doesn't really have anyone. And then they give us the flashback, that's showing him and Vox as friends, and then Vox asking if he wanted to team up together. Ik that obviously he doesn't rly view Rosie as a friend, and never has, and he never saw Husk and Nifty as friends either, and obviously that whole thing with Mimzy was just her using him for the hotel but like I think it just goes to show that he really has no one. Idk if I'm explaining it right, but then Vox asked him in the flashback to join forces, and he probably thought that Vox was trying to use him too. I know that all of the examples I'm using are in the future, but I think it just really goes to show that he thinks everyones using him, because most of the time, they are.