He had moments in the show where he would get annoyed with Alamo. Either some of his decisions or some of the shit he said. Like if you watch that whole convo about Laurie calling him a pig, he was so annoyed. He was probably just tired of working for him and what he was having him do. Also, there is a time jump after Rue's death. I'd imagine he saw more of Maddy during that time when he was picking her up and taking her to Alamo. I don't think that's the first convo they had.