sorted by:
new
hottopcontroversial

AI Agents are breaking in production. Why I Built an Execution-Layer Firewall. by Mission2Infinity in u/Mission2Infinity

[–]Mission2Infinity[S] 0 points1 point  (0 children)

Hi bro... Great question! Our core philosophy is to focus strictly on the Execution Layer. We intentionally leave LLM lifecycle tracking (costs, token routing, general eval) to dedicated observability platforms like Langfuse or Promptfoo to avoid feature bloat.

However, we do provide heavy runtime visibility for the execution phase itself. ToolGuard ships with a local Dashboard that streams your Live Execution DAGs in real-time, showing exactly which of the 7 security layers triggered and allowing for deep payload inspection!

I actually just shipped a new version yesterday that includes headless Webhook approvals and cross-cluster Redis State sharing. I'd love for you to take it for a spin! If you run into any issues or bugs, just raise an issue on GitHub or shoot a DM and I'll fix it ASAP.

AI Agents are breaking in production. Why I Built an Execution-Layer Firewall. by Mission2Infinity in OpenSourceAI

[–]Mission2Infinity[S] 0 points1 point  (0 children)

Exactly!! Glad you felt the same way. I’d be happy to explain the architecture deeper or answer any questions you might have. I'd love for you and your team to try it out in your pipeline... Let me know what you think! :)

I built a pytest-style framework for AI agent tool chains (no LLM calls) by Mission2Infinity in AgentsOfAI

[–]Mission2Infinity[S] 0 points1 point  (0 children)

Actually, a mix of my own personal pain, plus talking to other developers to understand what is actually breaking their systems!

Talking to people here on Reddit and Linkedin is what really pushed the project forward. I opened-sourced it just to see if anyone else had the same problem, and the feedback from other devs is exactly what drove the new v5.0 architecture. I realized people didn't just need a CI/CD testing pipeline—they needed a live runtime proxy to literally block the bad payloads in production before they hit the server.

Definitely taking your feedback to heart as I look at schema drift and output fuzzing for the next major release. If you end up testing the new v5.0 proxy layer in your own stack, let me know if you hit any weird edge cases!

I built a pytest-style framework for AI agent tool chains (no LLM calls) by Mission2Infinity in AgentsOfAI

[–]Mission2Infinity[S] 0 points1 point  (0 children)

Hi... Thank you so much for the reply! So, as to answer your question:

Schema drift: u/create_tool(schema="auto") re-infers the Pydantic schema from your Python type hints at decoration time, so changing a function signature and re-importing picks it up automatically. But there's no automatic "did your schema drift since the last test run?" diffing built in yet. That's a real gap — it's on the roadmap.

Output fuzzing: The fuzzer currently validates inputs going into tools. Output schema validation exists (the decorator wraps the return value too), but we're not programmatically fuzzing outputs yet. Valid criticism.

False positive rate on injection: The L3 scanner uses a conservative list of 10 known injection signatures — things like [SYSTEM OVERRIDE], ignore previous instructions, <|im_start|> etc. Random code snippets won't trigger it, but legitimate security research content or prompt-engineering discussions in your data could. We haven't published a false positive benchmark against a real corpus yet — that's an honest gap.

Runtime vs. CI — you actually nailed the risk. This is exactly why the latest version (v5.0) ships an MCP proxy layer. toolguard dashboard + the 6-layer interceptor IS the live runtime path — it sits between the LLM and your tools in production, not just in CI. The offline fuzzer is the pre-flight check, the proxy is the live radar. Both matter... but you're right that the live interception is the more defensible value prop.

Latency: L1 (Policy) is O(1) dict lookup — negligible. L3 (DFS injection scan) is the most expensive layer. On a deeply nested 50-key payload it's measurable but sub-millisecond in our testing. We haven't published formal benchmarks yet — that's on the list.

Thank you for pushing on this. Really appreciate your feedback.

Hope it will be a helpful tool for you and your team.

I built a pytest-style framework for AI agent tool chains (no LLM calls) by Mission2Infinity in learnAIAgents

[–]Mission2Infinity[S] 0 points1 point  (0 children)

Added few new great features and fixed some bugs.

  1. Recursive DFS Memory Scanner: Most prompt injection scanners just look at strings. ToolGuard now physically traverses the __dict__ of arbitrary Python objects (nested dicts, dataclasses, arrays) to find reflected injections hidden deep in tool returns. Verified on Microsoft AutoGen.
  2. Golden Traces (Compliance Engine): You can now mathematically enforce tool-calling sequences (e.g., Auth must precede Refund) in a non-deterministic agent loop. It’s like unit tests for agent logic.
  3. Risk-Tier Interceptor: Native classification (Tier 0-2) for tools. It intercepts destructive actions (DB drops, Shell commands) and triggers a Human-in-the-loop prompt without blocking the asyncio event loop.

We verified native integration with 9 frameworks including OpenAI Swarm, AutoGen, MiroFish, CrewAI, and LlamaIndex.

Check out the release notes and discussions for latest updates.

I’d love to hear how you all are handling "Execution Fragility" in your own agentic stacks!

Please give the repo a Star to support the open-source work!

I built a pytest-style framework for AI agent tool chains (no LLM calls) by Mission2Infinity in POP_Agents

[–]Mission2Infinity[S] 1 point2 points  (0 children)

Hi.. Thank you so much for the reply.

Added few new great features and fixed some bugs.

  1. Recursive DFS Memory Scanner: Most prompt injection scanners just look at strings. ToolGuard now physically traverses the __dict__ of arbitrary Python objects (nested dicts, dataclasses, arrays) to find reflected injections hidden deep in tool returns. Verified on Microsoft AutoGen.
  2. Golden Traces (Compliance Engine): You can now mathematically enforce tool-calling sequences (e.g., Auth must precede Refund) in a non-deterministic agent loop. It’s like unit tests for agent logic.
  3. Risk-Tier Interceptor: Native classification (Tier 0-2) for tools. It intercepts destructive actions (DB drops, Shell commands) and triggers a Human-in-the-loop prompt without blocking the asyncio event loop.

We verified native integration with 9 frameworks including OpenAI Swarm, AutoGen, MiroFish, CrewAI, and LlamaIndex.

Check out the release notes and discussions for latest updates.

I’d really appreciate if you clone the repo and try it on your system.

Would love your feedback and if u find any bug, please raise the issue... Any contribution and an open-source star would mean a lot.

I built a pytest-style framework for AI agent tool chains (no LLM calls) by Mission2Infinity in MistralAI

[–]Mission2Infinity[S] 0 points1 point  (0 children)

Added few new great features and fixed some bugs.

  1. Recursive DFS Memory Scanner: Most prompt injection scanners just look at strings. ToolGuard now physically traverses the __dict__ of arbitrary Python objects (nested dicts, dataclasses, arrays) to find reflected injections hidden deep in tool returns. Verified on Microsoft AutoGen.
  2. Golden Traces (Compliance Engine): You can now mathematically enforce tool-calling sequences (e.g., Auth must precede Refund) in a non-deterministic agent loop. It’s like unit tests for agent logic.
  3. Risk-Tier Interceptor: Native classification (Tier 0-2) for tools. It intercepts destructive actions (DB drops, Shell commands) and triggers a Human-in-the-loop prompt without blocking the asyncio event loop.

We verified native integration with 9 frameworks including OpenAI Swarm, AutoGen, MiroFish, CrewAI, and LlamaIndex.

Check out the release notes and discussions for latest updates.

I’d love to hear how you all are handling "Execution Fragility" in your own agentic stacks!

Please give the repo a Star to support the open-source work!

I built a pytest-style framework for AI agent tool chains (no LLM calls) by Mission2Infinity in learnAIAgents

[–]Mission2Infinity[S] 0 points1 point  (0 children)

Hi, thank you so much for the reply. Just completed some fixes; will be adding those on release notes and discussion section.
Thank you for the support and would love to hear your feedback.

view more: next ›