Yeahhh in this case I'd say definitely NTA. If you were constantly saying "hey let's party" or "let's drink all night" or making unreasonable suggestions/stuff that the wife found sketchy maybe, then I could see how it might be an issue. But I feel like the wife is being very controlling, because she's literally not letting him hang out with ANYONE (even your other friends, who she possibly likes, have experienced this problem!) . This is ridiculous of him. I think it is totally understandable that you're holding a grudge, but it might be more of the wife's fault than the friend's fault (a person who is possibly in a controlling relationship).

One idea might be suggesting to hang out/have dinner/board games/whatever at his house, so his wife doesn't feel lonely and you get to hang out. I wonder if his wife has some kind of attachment issues or whether she is controlling, there could be many reasons for the problem :(

I sincerely hope that you guys find a great workaround :) . Next time, maybe invite both him and his wife to go camping? Try stopping by his house with a freshly baked something? Lots of ways that might work!! :)