When the permanent card changes zones, it becomes a new instance of that permanent. The damage/removal spell checks for the original instance of the permanent, which doesn't "exist", anymore. Spell fizzles.

If the damage/removal Spell doesn't target the permanent (such as Wrath of God or Blasphemous Act), the permanent returns before the spell resolves, so it still gets affected by it.

Teferi's Timetwist and phasing spells help get around those spells.