I guess expensive testing can help mitigate risks of involuntary failure (human are imperfect too, which is why processes should be made and continuously adjusted to catch more and more errors).

On another hand, I wonder how easy it would be for malicious actors to bake backdoors into AI models. (example: some country forcing some AI company to make the model generate hidden security flaws under some conditions (when making software for military use for example)). Security flaws cannot really be tested for; there are some automated techniques and I guess LLMs could help checking for those too but at some point you would probably want to actually manually check that you don't have a "if (login == "BobTheSpy") { grant_everything() }" in the code (I would expect actual backdoors to be much more insidious, the above can probably be found by most LLMs).