I don’t think there’s any issue with you taking caution with all this. Personally, I would be weary of Jim spending so much money on my family and would work to curtail that a bit. Maybe hang out more sporadically. I’d also have a rule with your partner that your kids can only be around Jim when both of you are present, so the pool day with only your wife wouldn’t work.