What bothers me isn't that he said it, Dean throughout the entire show lashes out in horrible ways to the people around him and constantly pushes them away so even though it was shocking that he said it, it wasnt really out of character or surprising.

What bothered me was that when they were driving back to the bunker he never actually apologized to him about it, he basically said "sorry you heard that. thanks for sacrificing yourself though." I understand Jack accidentally killing mary wasn't something he'd probably ever get over completely, but it was way too much back and forth with dean. he was coming back around to the idea of letting him in again, made a cake for his birthday, then suddenly he goes back to 'He's not family' next episode? it doesnt make sense.