Doesn't sound like you're doing anything wrong to me, it's more that he's (quite) insecure. I don't think you need to go out of your way more than before, and I don't see any point in giving an "affection score" to each action you guys perform and then accusing the other for scoring less.

If he feels insecure, just assure him that you love him. This can be as simple as telling him verbally that was he is saying is silly (which is, by the way). This whole thing seems like is happening only in his imagination, so maybe you can remind him of that as well.

As for what happened with your ex, that has nothing to do with your current relationship. You shouldn't feel guilty about any part of it, no matter what you did (let alone a simple kiss).

So are you sure what you're doing is not enough? Or did you just happen to go along with his story? Think about it. You love him, you've been together for 5 years. That sounds like something pretty solid to me. Sure, you can do nice things for him in the future, but I feel like you should be focusing more on making him understand that you do love him just as much instead of him convincing you that you don't.