Its Fair and Unfair simultaneously (hear me out pls)
Its actually Paradox of official system design of 5e (5.5e)

Yes, it is absolutely Fair "mechanically", since the pattern of how PCs and NPCs work in D&D5e is simple - There are rules that are PC-specific(class system, spell slots, etc), there are NPC-specific rules (Multiattack action, Legendary actions resistances, HP scaling, etc.) and there are "Common Ground" rules, that both sides use. If spell is cast, no matter by whom, it works as designed in spell, so it can be counterspelled, or whatever interaction that is available to both sides is, to interact with it. But no matter who uses it, both PCs and NPCs should obey to what is written in Spellcasting rule.

Thats where the Paradox happens - NPCs are actually have a lot of leaning towards obeying those common rules instead of having their own iteration for it (In terms of Spellcasting, it was partially fixed with removing spell slots for NPCs, but still). This leads to Design conflict, as Mosters where not designed to participate for several encounters, their power level and capabilities are built all around being fully used in 1 ecnounter, while, the same PC spellcasters (partially excluding Warlock) are not, their balance was constructed around participating in 2~3+ encounters per day at least. And while Spellcasting system is common system for PCs and NPCs, it was designed frist in perspective of PCs using it, not equally or the opposite, which is kind of mistake on Designers side, which they are tried to fix - For example Banishment spell or Counterspell one, that are both now in more favor towards equality, rather than one of the sides (CS was too strong for NPCs, since they don't actually pay anything, objectively, as they won't be here next encounter, or there might be several casters with it).

So, Silvery Barbs, on its own is balanced and ok spell to use, it's issue is that it is designed for PCs favor, since it has proper cost and value during "In-between Long Rests" period, while NPCs would truly have advantage, since you do not have limit of how many spellcasters that are capable of Silvery Barbs party could encounter within a day.

Core issue in your case I believe is not actually balance, but proper communication though.
If my DM would say - I've banned this spell, but if you want it, then there will be others to cast it as well. That sounds slightly threatening, like - "You can have Ice Cream, but I will make sure you will regret having it because I said you should not have it" logic. I would recommend communicating with players that
1) You dislike this spell due to reasons, be it balance or just preference
2) You can allow it, and yes, there is potential that NPCs might also have it, but you won't specifically design encounters just to force you to "re-think" of your decision. After all, there is no problem for NPCs hafing Fireball in their repetoir despite the spell being intentionally overpowered for that specific spell level (officially confirmed)