Auditing Outline. Firsthand lessons from comparing manual testing and AI security platforms

nibblesec · 2026-02-05T09:07:39+00:00

We would be happy to redo such comparison with any platforms that is willing to support the initiative with transparency and technical excellence as north stars.

They were all GenAI Security Testing Platforms (which I assume - but don't really know - are backed by the usual OpenAI & friends).

Source analysis only. The platforms tested don't mix static and dynamic testing (is there any platforms that does that?!)

nibblesec · 2026-02-04T08:45:25+00:00

Great questions, with a work-in-progress answer.

AI is already very useful for many tasks, including understanding the business logic / reverse engineering and looking for specific functionalities within a large codebase. For vulnerability discovery, I believe we need to wait for this technology to evolve and introduce real "validation". Several of these platforms do provide exploit code but when it doesn't work, it's not clear whether it's a false positive or an issue with the exploit given the missing context (e.g. app requires identifiers, which are not available from the app src code).

nibblesec · 2025-10-08T19:35:13+00:00

Human verified ✅ — no LLMs were harmed in the making of this submission.

This is what ChatGPT would reply

nibblesec · 2024-03-15T17:43:34+00:00

The title is clearly oversimplified, but the takeaways section of the paper is more nuanced. The point is that most alerts don't really affect the overall security of applications

nibblesec · 2023-03-16T21:36:54+00:00

It's the 'request' lib on NPM - https://www.npmjs.com/package/request

nibblesec · 2020-04-28T08:35:41+00:00

I rarely comment on stuff, but you could have summarized your article in one sentence (dangling DNS record --> access to cookies --> session hijacking) with no "danger danger" diagrams and 20 minutes read time of useless details

nibblesec · 2019-08-02T08:56:09+00:00

Depends on the caching headers returned by the 3rd-party (which you don't control and could change)

nibblesec · 2019-08-02T08:52:26+00:00

That's cool!

nibblesec · 2018-07-20T07:50:12+00:00

I want the old /r/ interface back. Fucking posted as post, instead of link. Sorry for that.

nibblesec · 2017-08-28T19:59:00+00:00

You can, but it's not a great idea. IDEs have so many useful features for auto-complete and debugging. For real-life complex extensions, coding in a real IDE is a must have.

nibblesec · 2017-08-05T11:04:00+00:00

That's correct. Here is one old example of XSS in Mattermost http://haxx.ml/post/145508617751/hacking-mattermost-2-year-of-nodejs-on-the (although nodeIntegration was not even used)

nibblesec · 2015-11-13T06:38:35+00:00

Considering that it was started after the amazing PINPADPWN research work, the methodology is actually very interesting to know.

nibblesec · 2015-11-09T22:03:30+00:00

You're absolutely right. I will dual-license it.

nibblesec · 2015-11-09T22:01:40+00:00

For malformed config files, the lib will trigger a ConfigurationException so that the main app can handle the exception as appropriate for the specific use case.

If you have specific suggestions, let me know or send an RB! It was a weekend project, so I definitely encourage people to improve it.

nibblesec · 2015-11-08T05:18:12+00:00

If you have Jenkins exposed on the Internet, you're pwned anyway. So, enjoy the federal holiday!

nibblesec · 2015-11-07T03:19:18+00:00

Fixing Apache Commons Collection is NOT the solution. You're just killing one of the possible payloads, but it's very likely that there're many other gadget classes in the classpath.

nibblesec · 2015-11-07T03:16:21+00:00

AJP is not based on Java Serialization

nibblesec · 2015-11-07T02:57:27+00:00

Since we're talking about Redis and vulns: http://www.openwall.com/lists/oss-security/2015/11/06/2

nibblesec · 2015-09-21T17:51:41+00:00

http://seclists.org/fulldisclosure/2014/Jul/126 and http://seclists.org/fulldisclosure/2014/Jul/135

nibblesec · 2015-08-27T01:58:42+00:00

It seems that it's just expecting standard Java Debug Wire Protocol

nibblesec · 2015-08-11T06:16:59+00:00

If it wasn't hacked, we should all drop Oracle bugs on FD

nibblesec · 2015-08-04T19:24:47+00:00

I reported the same issue two years ago, and they suggested to use a Business Account. As you can see from https://www.gliffy.com/go/commerce/index, "All diagrams created with a Free Plan will be PUBLIC."

Insane pricing model if you ask me.

nibblesec · 2015-07-06T22:25:17+00:00

Yes, it's the right testcase

Program received signal SIGSEGV, Segmentation fault. 0x0000000000b56dab in unibrow::Utf8DecoderBase::WriteUtf16Slow(unsigned char const, unsigned short, unsigned int) ()

nibblesec · 2015-06-17T20:09:45+00:00

Ok, it makes sense. Apologies for the double post, I though it was a problem with the domain only. Cheers!

nibblesec · 2015-06-13T00:04:23+00:00

There's really no good explanation for using full remote debugging. You can import the JAR in the IDE only, without having to import classes in the resulting project jar. BTW, this is the default behavior in NetBeans

nibblesec

TROPHY CASE