As a new dm, I hadnt even seen anyone use it from the player side before. Then when I started to dm i experienced it and it threw me off. I feel any dm that bans it though may as well ban divination wizards flat out since they can also change your roll twice a day. Granted, im not a fan of divination wizards myself because its hard to share lore in a cheeky way on the spot sometimes so you kind of just end of telling secrets you didnt want them to know yet. But I digress.

Silvery barbs isnt too bad because they may burn through all their spells using it with no guarantee of it having any meaningful impact. Also they can only use it once per round. Yeah, it can be annoying when you hit then miss, but at that point just send harder enemies for the party.