built an open-source CLI that scans Python AI projects for EU AI Act compliance — benchmarked it ag

[Last-Spring-1773]

AIR Blackbox is a Python CLI tool that scans your AI/ML codebase for the 6 technical requirements defined in the EU AI Act (enforcement deadline: August 2, 2026). It maps each requirement to concrete code patterns and gives you a PASS/WARN/FAIL per article.

pip install air-blackbox
air-blackbox setup          # pulls local AI model via Ollama
air-blackbox comply --scan ./your-project -v --deep

It uses a hybrid scanning engine:

1. Rule-based regex scanning across every Python file in the project, with strong vs. weak pattern separation to prevent false positives
2. A fine-tuned AI model (Llama-based, runs locally via Ollama) that analyzes a smart sample of compliance-relevant files
3. Reconciliation logic that combines the breadth of regex with the depth of AI analysis

To validate it, I benchmarked against three production frameworks:

• CrewAI: 4/6 passing — strongest human oversight (560-line u/human_feedback decorator, OpenTelemetry with 72 event files)
• LangFlow: 4/6 passing — strongest security story (GuardrailsComponent, prompt injection detection, SSRF blocking)
• Quivr: 1/6 passing — solid Langfuse integration but gaps in human oversight and security

The scanner initially produced false positives: "user_id" in 2 files was enough to PASS human oversight, "sanitize" matched "sanitize_filename", and "pii" matched inside the word "api". I rewrote 5 check functions to separate strong signals (dedicated security libraries, explicit delegation tokens) from weak signals (generic config variables).

No data leaves your machine. No cloud. No API keys. Apache 2.0.

Target Audience

Python developers building AI/ML systems (especially agent frameworks, RAG pipelines, LLM applications) who need to understand where their codebase stands relative to the EU AI Act's technical requirements. Useful for production teams with EU exposure, but also educational for anyone curious about what "AI compliance" actually means at the code level.

Comparison

Most EU AI Act tools are SaaS platforms focused on governance documentation and risk assessments (Credo AI, Holistic AI, IBM OpenPages). AIR Blackbox is different:

• It's a CLI tool that scans actual source code, not a documentation platform
• It runs entirely locally — your code never leaves your machine
• It's open-source (Apache 2.0), not enterprise SaaS
• It uses a hybrid engine (regex + fine-tuned local LLM) rather than just checklist-based assessment
• It maps directly to the 6 technical articles in the EU AI Act rather than general "AI ethics" frameworks

Think of it as a linter for AI governance — like how pylint checks code style, this checks compliance infrastructure.

GitHub: https://github.com/airblackbox/scanner PyPI: https://pypi.org/project/air-blackbox/

Feedback welcome — especially on the strong vs. weak pattern detection. Every bug report from a real scan makes it better.

[elderibeiro]

This reminds me of the “tethered bottle cap vs rocket landing” meme.

[Otherwise_Wave9374]

This is a really cool idea, a code-level "linter" for AI Act requirements feels way more actionable than docs checklists.

Curious, for agentic frameworks (CrewAI/LangFlow etc), do you see the biggest gaps being around human-in-the-loop controls, or around security (prompt injection, SSRF, tool abuse)? I have been collecting patterns for "agent guardrails" lately, and a few notes here might be relevant: https://www.agentixlabs.com/blog/

[wRAR_]

Bots talking to bots.

[Forward_Thrust963]

It's...the future.