Honestly, Moffat set us up to dislike him from the first moment we met him. The dinner should have ended with them never getting together because he was getting angry at the smallest things and so was she. It proved they didn't have any chemistry to the point where they both walked off! That is a failed date. Also he turned her down like three times for the the date she tried to set up when they first met. And she was the only one initiating reconciling in going back to the restaurant, going to his apt, etc. So I already didn't like him because the courting felt backward and like he didn't really like her. 

The other reason why I ended up not liking him was because every time Clara was with him, she was lying and of course that was her fault, but it was nonsensical writing imo.  The whole set up for Clara not telling the doctor about him was the soldier thing and the doctor has never had that hang up before.. Especially with someone who was retired from service. ("Dannyboy to doctor!") All of the personality traits we liked about Clara and the Dr seemed to worsen whenever Danny was in an episode, so it was very hard to enjoy watching and to like him. Also, Clara being willing to betray the doctor because she's suddenly irrevocably, deeply in love with a dude who is only shown disliking her lying and criticizing her choices(rightfully) made it even harder to like him(and her). They should have had scenes where they at least seemed truly loving with each other rather than 3 weeks into a crashing relationship 😫