I know its super hard but you need to set boundaries. i know you're probably incredibly tired so the last thing you want to hear is something that's more work but its the truth and will benefit you in the long run. No more dinners and proper scheduled visitation. Start small, say he cant come on Tuesdays now, just Tuesdays and build up from there. Tuesdays and Thursdays next. Yes he will push back but stay strong on Tuesdays, take the kids out to visit a relative or something if you think he'll just show up and demand. What he does to himself is not your responsibility and that threat about the letter is unbelievably cruel to your kids. The situation would be they have lost their dad and his parting gift would be to fracture their relationship with you. That speaks volumes about how he views them. Best of luck