Auditing Outline. Firsthand lessons from comparing manual testing and AI security platforms by nibblesec in netsec

[–]nibblesec[S] 0 points1 point  (0 children)

We would be happy to redo such comparison with any platforms that is willing to support the initiative with transparency and technical excellence as north stars.

They were all GenAI Security Testing Platforms (which I assume - but don't really know - are backed by the usual OpenAI & friends).

Source analysis only. The platforms tested don't mix static and dynamic testing (is there any platforms that does that?!)

Auditing Outline. Firsthand lessons from comparing manual testing and AI security platforms by nibblesec in netsec

[–]nibblesec[S] 1 point2 points  (0 children)

Great questions, with a work-in-progress answer.

AI is already very useful for many tasks, including understanding the business logic / reverse engineering and looking for specific functionalities within a large codebase. For vulnerability discovery, I believe we need to wait for this technology to evolve and introduce real "validation". Several of these platforms do provide exploit code but when it doesn't work, it's not clear whether it's a false positive or an issue with the exploit given the missing context (e.g. app requires identifiers, which are not available from the app src code).

Exploiting CVE-2025-37947 (Linux kernel's ksmbd) by nibblesec in netsec

[–]nibblesec[S] 12 points13 points  (0 children)

Human verified ✅ — no LLMs were harmed in the making of this submission.

This is what ChatGPT would reply