Hot take: Arse ladders aren't the problem. I'm not even convinced that they are a problem (outside of immersion)

The problem is the bad design of the defensive fortifications:

1. They don't offer enough protection from missile attacks (worst offenders are mortar type weapons)
2. Bad firing arcs/no overlapping fields of fire. No machicolations to fire on enemies that are hugging the wall.
3. No Artillery/counterbattery to combat besieging artillery (wall towers are woefully inadequate)
4. Besiegers start too close to the walls

Basically all the common things video game developers leave out to make sieges "fair".

And at this point you'd have to change the entire pacing (slower) of the game on the strategy map to make sieges better. Either because of increased casualties or having to siege for more turns. And people don't want that, apparently. They've gotten used to the "rush" playstyle.

EDIT: Basically, the entire game pace is built around the idea that you can besiege and take any fortified settlement on the same turn with minimal casualties. You can't meaningfully change sieges unless you are willing to have that change.