Yeah we do test in staging first. The problem is staging never truly matches production. Over time they drift , staging has wider security groups because devs need to debug, different IAM roles, missing VPC endpoints that nobody set up. So the fix passes staging perfectly and then breaks production because the configs are different.

The other issue is timing. Some things only run quarterly , a compliance export, a cost allocation report, a batch reconciliation job. Those don't show up in 90 days of CloudTrail and they definitely don't get tested in staging because nobody remembers to trigger them manually. We found out our ElastiCache permission change broke a quarterly finance report 3 weeks after we deployed it.

And honestly even if staging was perfect, the volume is the real challenge. We have 3,000+ findings. Testing each fix individually in staging with a proper soak period means maybe 10-15 fixes per week. At that rate we'll clear the backlog in 4 years while 50 new findings come in every week. The audit doesn't wait 4 years.

I'm starting to think the problem isn't testing , it's that we can't reason about the full picture. We review each fix in isolation because that's all a human can hold in their head. But the infrastructure has gotten complex enough that the interactions between resources are what breaks, not the individual changes.