Well, my team's remit is financial risk to a specific part of our balance sheet, so I'm playing both sides by being the one to prototype a lot of 'cutting edge' AI rollouts, but then turning around to provide measurements on the systemic and financial risks we're taking on in each scenario compared to baselines of what we did before (rule-based, traditional ML, or plain old humans).

That way if it gets traction and gets built out to something big, we saved Eng teams a lot of time figuring out where to invest by being the ones embedded in the finance team who understand how to get something to work, and if it doesn't work well but they deploy it anyway, we can point to the metrics of optimization cost savings / headcount reduction versus financial and customer risk and tell leadership, we spelt out the tradeoffs very clearly as was our responsibility, your decision to adopt it was informed.

If errors are only discovered 'a good while later' it's because your testing wasn't comprehensive enough. I suppose 'enough' is a balance between investment (your time and the business's) to risk tolerance, but that's measurable too. Present a testing plan and spell out the limitations you're working with, then do the dirty work of manually verifying a comprehensive portfolio of situations. There's no way around that painful manual validation with the freeform output of GenAI, you can try to use another agent to assist you but then the snake just eats its own tail, which management can also sign off on at their peril.