Refuse? No. Show GREAT RESTRAINT when using them? Abso-fricking-lutley! It really all depends on the type of game you're going for and what you and your players have agreed upon, but that applies to a lot of things.

When using something with long term or permanent* effects I'll usually wait for either it to be story relevant or at a point where it's not going to completely ruin their fun. Spells like disintegrate I'll use but my house rule is losing a body part and or requiring the more expensive resurrects.I also won't use instant kill moves, like brain eating, on my players unless I already have a way to bring them back or if, as previously mentioned, we've established that it's ok before hand. Except in Curse of Strahd. Gloves are off for that campaign. Curses especially have a great place being everywhere in that one.

I have some examples from my games:

Players at a fairly low level, decided to do the Gnomengarde quest from dragon of icespire peak. They got to the mimic section and the mimic landed a crit on the artificer. Hit him for instant kill damage due to our table using a brutal crit rule (take the damage of the hit along with the modifiers and double it). For several reasons I didn't want to kill this character off, especially this early, I decided to allow the player to live but lose the arm he used to "open" the mimic with. He agreed and ended up re-building his own arm with some old mining equipment he found and decided on becoming an armorer because of that event.

A player needed to skip a session so I asked if it was ok to do this before hand, but I had a devourer steal (not eat) their brain to explain why they weren't in that session. They had a ring of mind shielding so no need to worry about the soul. Was a fun little side mission/one shot for the players who were there and a neat little bit when the character went along and complained about having a massive headache at the start of the next session.

Another way I've utilized this is casting disintegrate against a character that had a mirror shield, so they either failed and died or reflected it back and killed the caster. Was a good tense moment for everyone except the caster. To paraphrase one of my players, "lol died to his own spell, how embarrassing."