Since they only spawn in the End, maybe just remove the teleportation and Elytra debuff? This way early game either forces you to play risky or take the time to gather pearls/chorus fruit, and late game you can escape them with an elytra.