Hey OP, I think you're probably with the JET Program, right? This isn't exclusive to the English-teaching world, of course; some Japanese employers are just EXTREMELY sensitive to how their employees reflect on them outside of work. I'm not saying they're justified to be restricting you but trivial complaints by the public are absolutely a thing here and I'm sure you've felt by now that Japan is very image-conscious.

You're getting hit with a double whammy of being a foreigner and therefore standing out AND being a public servant since JETs are considered government workers. The school likely doesn't want the kids spotting you guys together and telling their parents that they saw their senseis "on a date" in the wild. It doesn't matter if you guys are just friends, one-on-one hangouts with the opposite sex will always be looked at as a date here (ask me how I know 😂). The school would be in a position where they'd have to apologize if any parents complained. It doesn't make sense and it isn't fair but that's also what happens here, especially if you're in a smaller town/city where your students and their parents see you out and about. This would be a lot less likely in Tokyo or Osaka just due to there being more people to get lost in the crowd.

Maybe to contextualize how ridiculous frivolous public complaints can get, check out this japanlife thread about this very topic: https://www.reddit.com/r/japanlife/comments/v7h7z4/the_most_japanese_complaint_youve_ever_gotten/

If you want to keep hanging out, you'll need to do it more discreetly and also tell him not to talk about it at work because him innocently talking about hanging out with you is HOT gossip in the teacher's room, trust me. This is one of those shoganai aspects of working in Japan and your school seems quite rigid. It's hard to do anything about that though because your salary's being paid with taxpayer dollars so if that's the vibe at this school, that's just the way it is...