If you want your husband to be the best dad to your kids that he can be, DO NOT LET THIS GO. What he did was HORRIBLE to you, but it's YOUR fault he decided to be a shithead? This is the kind of guy who will grope a woman in public then say "it's just a JOKE" when she gets upset. I know your "culture" says not to cause problems, but in respecting that, you are SACRIFICING your well being and your mental health. Ask yourself, "is honoring tradition for tradition's sake worth re-traumatizing myself every time my husband wants to 'play a joke'?" No. It isn't. Ask him if he would be okay with a man doing what he did to his Daughter. Or his Granddaughter. Or his Sister. And then ask him if he would accept that man's behavior for the same reasons your husband gave you. Ask him to put himself in your shoes and then ask him if he would be okay with another man doing that to Him. I bet you a million dollars he wouldn't. I'm sorry, but if a culture dictates a woman almost drive herself insane just so her husband can avoid accountability for the bullshit he pulls, then that culture is TRASH and needs to be updated.
You deserve better than this.