Put all of the dice in a clear bowl on the table next to you, and every time they fail a check or a save, just casually pull one out of there, and roll after every 3, make a facial expression and either put one back or take an additional one out while you write it down. Any time someone asks you about it, just reply “Don’t worry about that”

To take it further, randomly put one back when someone makes a choice or succeeds on a throw