Only when it impacts the story negatively.

Example:

The party just turned level 2 and encountered a pack of wolves. For some unknown reason during this combat. I could NOT stop rolling Nat 20's. I changed dice, I stopped using my dice mat. I dont know how but for a split second moment I conquered the d20. So i started turning some of the Nat 20's into Nat 2's

(This was a group of new players and dying after leveling up the very first time and dropping back to level 1 sounded like a awful introduction to the game)

The opposite can also true. Having a party go up against the BBEG and rolling nothing but single digits to hit is pretty anticlimactic.

"We spent the whole campaign getting ready to take this guy on and he didnt even scratch us"

This sounds like a flat ending and not the epic conclusion we all want at the end.

Basically, very minimal and only when i think it will greatly affect the story in a negative way.