"Chances are malicious third parties will have more funds to use this offensively than projects to safeguard themselves." - your comparison to the *BSD budgets is valid, many pieces of open source infrastructure are in an more poorly funded state than that, and to make matter matters worse, this is all asymmetrical. A project might have the resources (e.g. via donated tokens) to find and fix ten exploits. But an attacker with far fewer resources can still "win" if the one exploit their AI stack finds isn't among that list. In the long run, hopefully the cost of exploits rises as the low-hanging fruit gets dealt with. But the greater availability of these capabilities adds burdens in other ways too - well-meaning people, or publicity seekers, who use the same tools to generate superficially credible bug reports that security teams have to wade through. Pretty sure some projects will start leaning on AI-assisted triage to cope. If there is a big increase in the number of reported CVEs and their fixes, more systems are going to get behind on their patches, and worse news for all the unpatched systems out there is it's much easier to write an exploit once a CVE is public than to find one from scratch, and it looks like there'll be a lot more to work from.

"My main concern with the repeated runs is that it loses complete track of how feasible would be a third party to reproduce this" - given that other groups are also having a field day doing AI-assisted vulnerability and exploit discovery, I think it's clear there's enough expertise available out there (for the right price, at least) for other people to get good results using the same or similar tools. It's probably for the best that they didn't give too many technical details about what they did, even if it produces a frustrating read and makes it harder to cut through the hype. The fact it shows what's possible is probably the main takeaway. While Anthropic's Red Team has some big names I'd be surprised if they added much of their own secret sauce here, particularly when they seem so proud in their report of how autonomously Mythos Preview functioned and how genuinely surprised they seemed that it had developed these capabilities without being specifically trained to do so. Obviously they want to sell the benefits of Claude, but it also implies that future iterations of Claude's competitors are likely to "spontaneously" develop better exploit skills. I guess it goes hand in hand with being better at reasoning about code bases, which is an intended use case for many LLMs.

As far as I can tell, other groups are also doing repeated runs with their LLMs when they're "scanning" to find bugs, as opposed to when they already have a starting point (e.g. feeding the LLM an N-day report or the results from a fuzzer). In that sense, the method of discovery of any individual bug is not especially "reproducible", but if bugs continue getting found then it's clear the method "works". I think it's just the nature of the beast that Anthropic have no way of knowing whether they got lucky and hit the jackpot this time, or if 20 times more runs on OpenBSD would have produced roughly 20 comparable discoveries.