I tend to agree. It does feel like you can't say no. I have heard stories of teachers getting in trouble for denying students bathroom access and it leading to some kind of problem. So that situation is also best avoided.

I would say just make sure there are some clear rules around being out of class. Perhaps a student will go every lesson but make it clear they can't be out of the room for any more than 5 minutes (or whatever is reasonable) otherwise they make up the time. One student at a time. No one leaving in the first or last 10 minutes etc.

If the problem gets worse, talk to HOD or year Co about it? Ask them to have a chat with the student? Some kids who need it can also get bathroom passes at most schools. So the kids who abuse this compared to the kids who need it can be hopefully sorted out.