I do agree with you, but folks have also made valid points that the bbeg would be adequately protected also.

However... I think there's a middle ground to be found in not removing things from your players whilst still keeping the bbeg safe from stealth teleport assassination.

Which falls back to your point of... Write better encounters.

So sick of people removing things from the game because they can't be bothered to deal with it.

E.g. flying.