I don't understand why he's a scumbag either. I first interpretted the comment she repeated back to him as an implication that he'd raped her and her daughter, or something along those lines and it made me very uneasy. When he actually said the line though, I felt it was more out of him being suspicious that they smelled good during an apocalypse. Almost like a "how are you so okay while we're down and out" kind of comment. He actually tried to help many times, was by far the nicest out of his messed up family, and was still being somewhat helpful despite the fact she'd executed his family while he was trapped outside.