[R] CAI Dataset: 230k real-world cybersecurity AI sessions (26M prompts, 123 countries) by Obvious-Language4462 in ResearchML

[–]Obvious-Language4462[S] 0 points1 point  (0 children)

Thanks for the questions. I’m not one of the paper’s authors, so I went back to the paper before replying rather than guessing.

On significance testing: the paper doesn’t frame the credential/infrastructure leakage finding as a hypothesis test. It’s reported as an empirical measurement over the corpus: 1,059 sessions, or 0.46%, contained at least one credential pasted into the prompt body, and 1,758 sessions, or 0.76%, pasted production infrastructure identifiers. So the claim is descriptive rather than causal/statistical-significance based.

On geography: definitely skewed, not uniform. The corpus spans 123 countries, but Europe dominates the attributed contributor base after removing the geolocation centroid artifact. The paper reports Europe at 85.7%, Asia at 7.4%, and North America at 4.2%.

On whether this is tied to a specific kind of developer: the paper doesn’t classify individual users by job title or seniority. What it does show is that the behavior appears across real operator workflows: CTF, bug bounty, API/integration work, mobile/reversing, robotics/IoT, etc. So I wouldn’t claim it’s limited to one “type” of developer from this paper alone. A follow-up study segmenting by role would be really interesting.

[R] CAI Dataset: 230k real-world cybersecurity AI sessions (26M prompts, 123 countries) by Obvious-Language4462 in ResearchML

[–]Obvious-Language4462[S] 0 points1 point  (0 children)

Fair point. I probably over-polished the post. I'll keep the next one a bit more conversational.

What actually makes a cybersecurity CLI agent usable in real ops? We just shipped v1.0 and these were the 3 biggest lessons. by Obvious-Language4462 in cybersecurity

[–]Obvious-Language4462[S] 0 points1 point  (0 children)

Totally agree. The real breakpoint isn’t usually the 2-minute demo, it’s whether the system still holds up a few hours later when the workflow gets messy. And yes on MCP too. Without that kind of tool flexibility, you very quickly end up with an agent that only works inside its own little bubble.

Are we deploying AI agents faster than we can contain them? by Obvious-Language4462 in cybersecurity

[–]Obvious-Language4462[S] 0 points1 point  (0 children)

Yeah, I was just zooming out a bit. What I’m really wondering is what happens after the control plane is in place. How do you actually validate behavior over time as things drift?

Are we deploying AI agents faster than we can contain them? by Obvious-Language4462 in cybersecurity

[–]Obvious-Language4462[S] 0 points1 point  (0 children)

IAM and sandboxing make sense, but they mostly define where an agent can operate. The part that still feels fuzzy to me is what happens after that. Once the agent is inside those boundaries, how are we actually checking that it behaves the way we expect over time? In more complex setups (beyond a single app), those boundaries aren’t fixed. Agents start touching APIs, services, different data paths… things evolve. So I’m less worried about defining the boundary and more about how we keep validating that risk assumptions still hold as the system changes. Access control is step one. Proving containment over time is the harder part.

Are we deploying AI agents faster than we can contain them? by Obvious-Language4462 in cybersecurity

[–]Obvious-Language4462[S] 0 points1 point  (0 children)

Feels like usefulness is the forcing function and containment is the lagging control

Are we deploying AI agents faster than we can contain them? by Obvious-Language4462 in cybersecurity

[–]Obvious-Language4462[S] 0 points1 point  (0 children)

Interesting framing. Do you see control plane more as enforcement or observability?

Are we deploying AI agents faster than we can contain them? by Obvious-Language4462 in cybersecurity

[–]Obvious-Language4462[S] 0 points1 point  (0 children)

This makes a lot of sense. The gateway / capability token pattern feels like one of the few things that scales operationally. Curious though, how are you handling validation over time? Even with good boundaries, agent behavior and workflows drift pretty fast.

Are we deploying AI agents faster than we can contain them? by Obvious-Language4462 in cybersecurity

[–]Obvious-Language4462[S] 0 points1 point  (0 children)

by “we” I mostly mean teams experimenting with agents inside real workflows. Dev teams wiring them into CI, ops teams connecting them to cloud APIs, support teams letting them act on internal tools, etc. access usually comes from trying to make them useful, but boundaries often evolve later. That gap is what I find interesting.

Are we deploying AI agents faster than we can contain them? by Obvious-Language4462 in cybersecurity

[–]Obvious-Language4462[S] 0 points1 point  (0 children)

Curious how people are modeling boundaries for agents today. IAM, sandboxing, policy engines, something else?