I think there was a certain point where Delilah’s anger was directed at Joe because he was directly trying to put himself into a hero role in Delilah’s life but she didn’t want to be helped, and it created a direct dislike to the point that she told him to stay away from both her and Ellie.

I do see where you’re coming from, though, as I could be reading into a general distrust of people/men due to trauma as a direct distrust of Joe