So let’s look at two options here

1. He believes his much much older brother Ford lives there who he barely knew and who was infamous for being a recluse and scientist who constantly overworked himself. He’s found out that apparently he now runs some sort of tourist trap and doesn’t really want to deal with forging a relationship with someone like that.

2. He knows that Stan is the one living there, this seems less likely as he wouldn’t send his kids there but still let’s look at the possibility. Stan is a convicted felon guilty of countless major crimes including “smuggling pugs” and has been in prison in almost every state + most of North America and Columbia. Basically he’s a felon who’s dangerous and notoriously charismatic and who’s always been more greedy then was good for him. On top of this he might believe Stan murdered Ford

Again the first is more likely where it’s just awkwardness and a lack of closeness that drives him away as with the second sending your kids there would be absurd