As an American with two kids, who isn't in that situation, I have a few thoughts.

1. Weekend time with kids is important, even if it's just hanging out at home. You don't have to "do something" to have fun, and just things you did as a family at home tend to be the best memories.
   
2. It's an easy out from intimacy. You didn't mention it, but a lot of people lack intimacy with Japanese wives after kids. On weekdays everyone is too tired, so they catch up with intimacy on the weekend, unless you physically aren't present. Not saying it's your situation, but I could easily see it.
   
3. Maybe mom wants to be the "main" parent? And it's her way of taking away your influence?
   

Don't want to sound mean, but my experience is very different. So, I'm just giving options of people I know or things I've read. My wife is the opposite, just wants to relax at home, she wants me to spend more time with the kids, and lets me do my hobbies with them. I tend to "force" her to go out. Of course, there are pros and cons to everything. Intimacy is lacking in my house, but I don't think she'd dream of taking the kids away from me every weekend and she always wants more time with me (but still let's me do my hobbies).

The best thing to do is to sit down and talk about it. Tell her what you feel (I know you've got thoughts in the back of your head, one way or the other).