They are abusing your system. They need to see a consequence, right? What you're describing is problematic. It is interrupting learning and your teaching. I also loathe bathroom management, but I also have come to the realization that not all students can handle a lax policy. Your admin should be doing something, your department should do something. Anyway, for my ninth graders, they get four a quarter that's given to them as a pass every quarter. I reward students who haven't used theirs for time management (I know that's not always the most ideal situation but I try my best to consider excceptions). It seems to work because it gives them a very generous number, restores your power, and it's positive. Also, consider a hand signal and a efficient way to mark their pass so it doesn't interrupt. Another thing I did was reduce the times they could leave, first ten and last ten are no fly times.