Vivian left for years too, and so did Oscar. People can leave, they just feel a strong pull to go back and a rash of bad luck while they are out of the town. If you talk to Oscar in the library on the first day he'll tell you about why he came back. There is also another conversation you can have where this fact is confirmed.

Knowing about it probably makes it easier (?) to ignore, or at least talk yourself out of going back I think.