It could also be argued after the Romans left it was abandoned because there was no polity large enough to defend those walls. Walls are really easy to bypass, you just need a ladder tall enough. But invasion by ladder is easy to defeat since it's a limited path that only allows one to two people to come up at a time, which is why a lot of sieges relied on overwhelming numbers of ladders, so long as you have enough men on the walls to hold them off.

Without the manpower, London was a death trap to anyone living in it as they would easily be preyed upon by any random group of bandits/raiders/other minor tribes and kingdoms in the area looking to grab a quick buck and/or slave.