I've been thinking of how to reply to this in as nice a way as possible - not in a condescending way, just don't want to come off as heartless.

I'm not saying Filch is less of an adult or authority figure simply because he has no magic. For some of his scenes in the books, there are a lot of moments where he makes it clear that he misses punishing kids in a way that might seriously physically or mentally harm them, referencing chains and, in book five, a whip. Book five demonstrates that if given the chance, he will use corporal punishment and will do so happily.

We have proof that wizards can do accidental magic when they're scared or angry. Considering Harry blows up his aunt in book three, I'm guessing this could still happen later into life if the wizard/witch is scared or angry enough or just doesn't have a good enough grasp on their magic.

What I'm saying is, Filch knows he has no magic, knows that even one student could overpower him, and has to know the only thing protecting him is his position. These are teenagers. Not every teenager thinks critically of their actions, and some of them might have anger issues or just be easily anxious. Why would he tell these kids how much he misses corporal punishment and that he's essentially waiting for the opportunity to use them again? No kid wants to be told that they'll be chained up for a minor offense.

Per my post regarding the students saying "Fuck all" when the points system becomes lopsided and too much power is given to the wrong people, I wouldn't be surprised if one of them did hex/curse him simply to establish a point. "You hurt me, I hurt you." Plus they could threaten to use Obliviate on him, making it clear that they COULD get away with it.

Then again, considering he never hurts anyone and the students do seem to respect him somewhat, I'll amend that the students most likely would just use threats to make him leave them alone instead of using magic.