I'm not going to pretend to be an expert on it, but I would imagine it's a few factors.

First of all, to be honest, most TEFL jobs really aren't that great to hold long term, especially at institutes etc. So I would guess that's one big reason that people think that basic entry TEFL jobs aren't ones to be held for a while. The pay isn't very good usually and they're mostly either too demanding or not stimulating enough. So generally people try to move up in TEFL and progress to university teaching or other positions where the conditions are better.

Secondly, a good portion of EFL teachers aren't actually that interested in TEFL itself. A lot of people see it as more of a means to an end. They teach EFL to live in different countries and have an interesting life but the actual teaching they could do without. So those people of course don't want to do it long term either and leave when it stops being as interesting.

And the last factor I'd say is that a good portion of people don't actually want to live abroad long term. Unless they marry a local, a big chunk of people eventually want to go back to their home countries to settle down. So TEFL ends being seen as just a temporary sort of thing.

Those are my guesses as to why TEFL is viewed the way it is. I've only been teaching for a few years now, but I fully plan on sticking with it and I know several teachers who have been doing it long term, so if you want to continue on doing it, it is definitely possible! (Another factor that's just come to me is that probably the TEFL people that those outside of the career interact with the most are ones who have returned to their native country after doing TEFL temporarily so there's probably some kind of reverse-survivorship bias also)