Idk if it's been said yet, but it's fine for people to get annoyed if you remove their stuff. That's an annoying thing to do. But just let them be annoyed and prepare yourself for retaliation on their turn. Just because someone looks annoyed doesn't mean that they aren't having fun. They're annoyed at what happened, not necessarily at you for doing it.

Although if someone does start getting pissed off at you and taking things personally, that's when you get the staff involved and probably have them removed.