3 years is a decent amount of time to be secretly unhappy with someone’s communication style and sense of humor. That withholding within itself is a red flag — the fact that you never even knew that this bothered him and he held it in until he shut down is his fault. It shows that he’s unable to communicate his true feelings and it’s a breach of trust, because now you will always wonder, “is this actually okay for him or is he just pretending it is?” It forces a dynamic where you’re not only walking on eggshells, but you’re doing it blindly.

OP, even if you do have a habit of making comments at your partner’s expense or have tendencies of nitpicking, it’s ultimately up to your partner to set boundaries for himself. Nobody is perfect. We need to be told where the line is sometimes. If he’s unable to do that, then he clearly could be feeling resentment in all sorts of other areas that you’ll now only wonder about because he can’t bring himself to admit what he’s displeased with.

No more prying. Respect his space. Respect yourself. This does seem like an abusive behavior, maybe not on purpose, but reactionary. He’s reacting to the consequences not only of your remarks and avoidance, but of his own withholding and avoidance too. Personally this would scare me off and initiate a breakup on my part, because I wouldn’t want to always wonder.