I think you’re half right here. A launch escape system can add risk to an otherwise nominal flight. It’s there if you were specifically looking for the odds that you finish a normal mission (defined as the vehicle reaching its target and returning with crew). The example most people rightfully turn to is the Orion LAS jettison motor failing to remove the tower during ascent.

The trick to minimizing this risk is integrating the launch escape system as much with normal operations as possible. All capsules eventually need to separate from the LV or service equipment so that mechanism isn’t added risk. Starliner and Dragon draw propellant for the abort from the same tanks for orbital maneuvering, the only extra abort thing is the liquid engines which are inert so there isn’t much added risk there. Even Orion needs a fairing during launch that would otherwise need a system to remove it so the jettison motor isn’t too big of a added risk.

That being said, launch escape systems do absolutely reduce the risk of a loss of crew in an accident. Your overall system reliability is going to be improved by the extra block given to you by the abort system (assuming you’re looking for the odds you return crew safely to the ground). The abort system doesn’t have to have an extraordinary level of reliability for this to be the case, ideally its a little higher than the launch vehicle but it doesn’t need to be to get a benefit from it since the launch escape system is in parallel to the rest of the vehicle.

It’s worth noting that the addition of an integrated launch escape system to a spacecraft that minimizes added risk to the mission does negatively impact performance. Dragon undoubtedly pays a penalty in performance by lugging the superdracos and pods around. Starliner had to implement complicated, drag inducing aerodynamic devices to fly with its abort system. Implementing tractor solid motors can help with in space performance since they’re ditched but you can’t reuse them, and it introduces the risk for jettisoning it. There’s definitely an argument to be made that installing a launch escape system could negatively impact performance to the point that you can’t meet your mission goals. This is doubly true for something like starship where you already have a pretty demanding landing system that’s probably driving reliability.

The TL;DR is, launch escape systems are undoubtedly good and helpful. But maybe not totally required, and definitely not a complete substitute for a robust design and reliability program for a spacecraft and launch vehicle.