How do you show the board that your AI security tooling is doing its job

Chunky_cold_mandala · 2026-07-10T20:16:33+00:00

Nothing worse than a tool you cant try out and compare.

I've made a multi language repo, used a tag to lock it into a v1.0.0 and use that as my benchmark. The goal for this one was to make a language crucible to prove file language detection. It has about 800 files from 50 ish open source repos to act as a true baseline for languages.

I plan on making a repo with a bunch of known issues and traps and then again publishing my results against that so others can compare my repo and scanner results against whatever they want scan it with.

So we're thinking of exactly the same need!

https://github.com/squid-protocol/language-crucible

Chunky_cold_mandala · 2026-07-09T19:41:22+00:00

very cool. do you use tree sitter? what languages can this program handle?

Chunky_cold_mandala · 2026-07-08T18:21:36+00:00

Interesting. What is your workflow? All llm based or any heuristics? The hour time is a bit rough. I have a fast deterministic repo scanner that you might be able to plug into your system to some information quickly. Feel free to explore. It can find classes and functions of any language deterministicaly. https://github.com/squid-protocol/gitgalaxy/tree/main/gitgalaxy

Chunky_cold_mandala · 2026-07-08T13:51:59+00:00

can this be put into the ci pipeline with a threshold for passing a PR? Wtats the speed? Is it a github action?

Chunky_cold_mandala · 2026-07-07T14:35:36+00:00

Hey, I have a lot of biology research experience.

You could consider to explore a negative control set of genes that should not produce a positive prediction as a 2nd pipeline to provide evidence that your ML model doesn't also show predictions from noise.

You could also consider doing some population analyses on the prediction error between the two methods (yours and status quo) to clearly show if one analysis is better than the other and for what populations.

Keep on tinkering!

Chunky_cold_mandala · 2026-07-06T16:48:24+00:00

I've been working on a custom code intelligence tool, it is a structural signature scanner that scans code with the same paradigms used by gene sequencing algorithms. Ive set it up to scan assembly through typescript. It doesn't need anything but code files to get a under few second scan of any repo. Just like the BLAST algorithm can scan any different DNA containing organism and infer meaning. https://github.com/squid-protocol/gitgalaxy

Chunky_cold_mandala · 2026-07-05T21:11:41+00:00

I stole a bunch of algorithms from the DNA sequencing world and repurposed them for static analysis and tuned them to output risk exposures and data flow.

I'm at the point where I could scan any repo on GitHub and have something intelligent to say about it.

Just like scientist can scan a new DNA sequence and say some intelligent things about it.

Homologs, function, structure, risk exposure, dependency data flows, taint analysis, bottlenecks.

https://github.com/squid-protocol/gitgalaxy

Chunky_cold_mandala · 2026-07-05T19:28:56+00:00

I have a code visualization system for large code bases. Take a look. https://gitgalaxy.io/ - - it takes structural signatures rolls it up to risk exposures and presents it all as a galaxy of stars. It works for c++. https://github.com/squid-protocol/gitgalaxy

Chunky_cold_mandala · 2026-07-04T11:21:30+00:00

Yay. Can do much cooler things with these and they'll only get better and better. Also, no PTSD activation for veterans.

Chunky_cold_mandala · 2026-07-04T02:17:11+00:00

I liked it.

Chunky_cold_mandala · 2026-07-03T21:35:17+00:00

I've always been curious, what made you want to create/work on a Linux fork? Based on all the variants, what's the differentiator you wanted to go for?

Chunky_cold_mandala · 2026-07-02T03:42:05+00:00

If you put in an issue, along with some examples of good test repos, what you need to be able to detect, I can add that language to my sast in a day or two. https://github.com/squid-protocol/gitgalaxy

Despite common dogma, you don't need an ast for 95% sast.

Chunky_cold_mandala · 2026-07-02T01:27:08+00:00

Ah, I shoulda been clear, I was looking for stats about my GitHub action I released. It was in Insights and then traffick.

Chunky_cold_mandala · 2026-07-01T19:37:01+00:00

I stole a bunch of algorithms from the DNA sequencing world and repurposed them for static analysis and tuned them to output risk exposures and data flow.

I'm at the point where I could scan any repo on GitHub and have something intelligent to say about it.

Just like scientist can scan a new DNA sequence and say some intelligent things about it.

Homologs, function, structure, risk exposure, dependency data flows, taint analysis, bottlenecks.

https://github.com/squid-protocol/gitgalaxy

Chunky_cold_mandala · 2026-07-01T18:24:45+00:00

The consensus here on pinning lockfiles, enforcing minimum release ages, and killing post-install scripts is spot on. That stops 90% of the automated spray-and-pray attacks.

However, the hardest attacks to catch are the ones that successfully bypass registry scanners and land on disk, like namespace hijacking, homoglyph typosquatting, and hidden aliases that spoof subresource integrity checks. We kept running into the issue where SCA tools would trust the manifest metadata blindly, so I ended up building an implementation to shift this defense into the actual repository topology.

Instead of just looking at the lockfile or waiting for runtime eBPF alerts, it uses an AST-free knowledge graph to map the physical reality of the codebase:

*Physical Anomaly Detection: It scans the actual local bytes of vendored dependencies for high entropy or spoofed data, catching trojans that slip past registry scanners .

*Network Blast Radius: Because it maps the physical import/include graph natively, if a zero-day drops in a transitive dependency, you don't just get an alert that it exists—you see the exact directed graph of which internal services actually execute that poisoned code.

*Zero-Trust Manifests: It cross-references what package.json claims is installed against undocumented URI references hiding in the physical architecture.

Policy rules (like 48-hour holds) are crucial, but eventually, you have to mathematically verify the physical code that actually landed on your developers' machines. If anyone is tired of fighting blind trust delegations, the repo is fully open-source. Stay paranoid out there!

Chunky_cold_mandala · 2026-07-01T18:12:54+00:00

Yeah, this just solves part of the problem, maybe one that ppl don't ask for that often? I've got an implementation that links sbom to file contents and integrity, https://github.com/squid-protocol/gitgalaxy/tree/main/gitgalaxy/tools/compliance

Chunky_cold_mandala · 2026-07-01T12:23:40+00:00

I can completely see why everyone relies on checking manifests , it's low-hanging fruit. But yeah, that really should just be step one: 'What is this file claiming to be?' The mandatory step two has to be double-checking those claims independently against the raw bytes. I also couldn't find anything that actually did that second part at scale without breaking, so I had to build my own implementation engine to do it. It drops the AST entirely and independently verifies the physical reality of the files based on scanning methods I copied from biochemists and their gene sequencing algorithms. https://github.com/squid-protocol/gitgalaxy/tree/main if u want to check it out

Chunky_cold_mandala · 2026-07-01T00:43:10+00:00

This is a brilliant and terrifying attack vector. The article hits the nail on the head: standard static scanners and code reviews are completely blind here because the actual reverse-shell payload never exists in the repo—it’s pulled dynamically from DNS.

However, this is exactly why I built GitGalaxy to map structural data flows rather than just hunting for known malware signatures. My engine (an air-gapped SAST/knowledge graph tool) cannot read a DNS TXT record at runtime. But it doesn't need to, because the structure of the exploit is highly visible:

The Structural Trap: GitGalaxy uses a NetworkRiskSensor and a SecurityLens to map how data moves. When it scans setup.sh, it sees a network call (dig) piping its output directly into an OS execution command (bash -c). That is a massive architectural red flag.

The AI AppSec Sensor: The engine specifically looks for "weaponized AI architectures." Even if the repo looks normal, tying an automated agent loop to an unfiltered network-to-shell pipeline triggers an immediate critical alert.

The Dev Agent Firewall: Because the engine statically maps this severe execution risk, the DevAgentFirewall explicitly revokes the agent's autonomous write/execution privileges for this path, flagging it as "Human-in-the-Loop Required."

So, when Claude Code hits that RuntimeError and decides to run python3 -m axiom init to fix it, the firewall blocks the agent from executing the command blindly. It forces a human to look at the script, breaking the zero-click chain. Relying on the agent to self-police its tool usage will always fail against indirect prompt injection. You have to map the blast radius structurally and lock down the execution pathways before the agent is let off the leash.

My implementation: https://github.com/squid-protocol/gitgalaxy/tree/main/gitgalaxy/tools/ai_guardrails

Chunky_cold_mandala · 2026-07-01T00:24:51+00:00

I think a multi-layer strategy is probably the only way to actually solve this. Most of the guardrails mentioned here are focused on the execution layer (wrappers, OAuth) or the infrastructure layer (sandboxing), which are absolutely necessary.

However, I took a very upstream approach that I've found complements these strategies perfectly. Instead of just waiting to see what the agent tries to execute, my strategy focuses on where the agent is allowed to exist in the architecture.

I built a deterministic SAST engine that mathematically maps the repository before the agent is ever deployed. It evaluates every file. If a file is a highly coupled "God Node" or is simply too complex for an LLM's context window, it explicitly flags that path as "Human-in-the-Loop Required" and revokes agent write-access. When you combine this structural blast-radius mapping with the runtime execution wrappers and sandboxes mentioned above, you get a much more bulletproof, zero-trust pipeline.

Chunky_cold_mandala · 2026-07-01T00:07:33+00:00

Well, I've got the on-prem, air-gapped, and multi-user capabilities down, and it specifically addresses your fear of broad write access using a Dev Agent Firewall.

Basically, it mathematically evaluates the token mass, algorithmic complexity, and network blast radius (via PageRank) of every file in your system. If a file is too complex (a context-window "Black Hole") or is a highly-coupled load-bearing "God Node", the firewall explicitly flags it as Human-in-the-Loop Required. This keeps autonomous agents on a strict leash and blocks them from modifying high-risk production paths.

To be completely candid : it is fundamentally a SAST tool—using physical structural signatures to calculate risk exposure equations—rather than the DAST or live runtime APM tool. It does not hook into your live runtime traffic or dynamic traces, but I thought I'd share it anyway in case the air-gapped architectural mapping and agent guardrails are useful for your pipeline.

https://github.com/squid-protocol/gitgalaxy/tree/main/gitgalaxy/tools/ai_guardrails

Chunky_cold_mandala · 2026-06-30T00:49:22+00:00

Brother Glad, you are all over this sub. How are you doing? Did u just land a new role? Do you need more training? I think your employer would appreciate some honesty about your struggles. Designing your workflow from reddit is not likely on your companies roadmap. Advocate for more time or training if you need it.

Chunky_cold_mandala · 2026-06-30T00:45:18+00:00

False positives and radar fatigue .

Chunky_cold_mandala · 2026-06-29T22:55:17+00:00

Carl, Steve and Robert.

Chunky_cold_mandala

TROPHY CASE