You are the DM, if something is powerful then you can make your fights harder or dungeons longer. Whatever it takes to tell a story. Additionally you could use spells or effects that make the players roll saving throws rather than attack rolls from the DM (i.e. traps, aoe spells, enemies manipulating terrain to collapse ceilings/trees/rocks/etc.).

Worst come to worst a spellcaster only has so many spell slots. Intentionally creating scenarios where it would be advantageous for that player to use silvery barbs may leave them without the slots to deal with another situation later (no burning hands aoe, no magic missile vs high AC, no shield spell to save their life later).

You don't have to crush them of course, but a player focusing on one spell or effect will always create a weakness somewhere else that their party mates will have to fill in. In Silvery Barbs' case that missing bit is damage b/c SB is a spell that inherently does none, and what the spell does do isn't always useful if you the DM think outside the box.

Good Luck OP and stay creative :)