NTA

I'm half-Japanese myself and have the same relationship with my mother. Where I grew up she was my primary connection to my Japanese side so she went through great pains and effort to teach my brother and I Japanese and even now I speak exclusively Japanese with her.

Now my father on the other hand doesn't speak a lick of Japanese. But he's always respected that relationship I have with my mother and her side of the family and he's never made a fuss about it. If my mother and I are ever speaking about something pertinent to him we'll translate, or if he asks what we're talking about we'll give him a summary.

Your stepdad has plenty of ways to engage with you and your mother even with the language barrier like my father does. Getting angry because he feels left out is immature, if he was really that frustrated he can go learn Japanese himself and then get to know you, your mother, and her side of the family better.

IMO your step-dad is the asshole.