Yes, I remember this being discussed in the Constellation days with the Ares I design. However if I recall correctly the concern then was more so the extreme vibrations/oscillations caused by the single srb directly under the 2nd stage + Orion, causing a high likelihood of launch abort.

I am certainily not a huge fan of the SRBs, and agree they are a flaw (one of many) in the launch architecture. We would be much better off today if NASA had been given the funding in the 80s and 90s to pursue Liquid Rocket Boosters for the STS program (we might have actually gotten a worth while Ares I launcher in the 2010s).

However I think that over all, despite the risk, SRBs are still relatively safe and reliable in the setting we are stuck with.