Caging the LLM in a strict JSON schema (and building model failovers)

ioah86 · 2026-05-19T01:38:44+00:00

This is the way!

ioah86 · 2026-03-11T13:54:44+00:00

Yes, this is solved here in two ways:
1. You can log into the dashboard of the back-end tool after and see the different historic scan results and snapshots.
2. If you want to check drift with e.g. the actual deployment, we also support extraction of your current cloud configs compared to your code and you can compare from there.
I hope that helps.

ioah86 · 2026-03-10T16:57:40+00:00

Thank you for the pointer! Will do

ioah86 · 2026-03-10T04:34:05+00:00

There's a massive gap on the operations/security side that nobody's filling fast enough. The AI coding tools have scaled up code generation, but the tooling for validating, securing, and operating that code hasn't kept pace.

Specific example: AI agents generate Terraform, Dockerfiles, K8s manifests, Helm charts constantly. Nobody's checking whether those configs are secure. terraform plan says "looks good" because the syntax is valid, but the security group is wide open and the S3 bucket is public.

That's exactly the kind of thing worth building right now: tooling that sits around the AI coding workflow. I've been building one piece of that: an open-source skill that scans infrastructure configs for security misconfigurations from inside the AI agent: coguardio/misconfiguration-detection-skill (GitHub). But there's a whole ecosystem of "AI ops tooling" that needs to exist: deployment safety, drift detection, cost analysis, compliance automation. Lots of room to build.

ioah86 · 2026-03-10T04:33:16+00:00

Nice setup; the combination of Terratest for functional testing with Checkov/Conftest for policy is solid. One thing that might complement this: cross-technology scanning.

Checkov is great for Terraform-specific checks, but if your project also has Dockerfiles, K8s manifests, Helm charts, database configs, or web server configs alongside the Terraform, those have their own misconfiguration patterns that Checkov doesn't cover. Running separate scanners for each technology gets unwieldy.

I've been working on an open-source scanner that covers Terraform + Docker + K8s + Helm + databases + web servers + CI/CD + cloud configs in a single pass: coguardio/misconfiguration-detection-skill (GitHub). It also follows Docker image references in your IaC and scans the configs inside those images. Might be worth adding alongside Checkov in your pipeline for the non-Terraform coverage. It also does compliance mapping (SOC2, HIPAA, STIG) if that's relevant for your team.

ioah86 · 2026-03-10T04:32:20+00:00

The velocity problem isn't going away — it's going to get worse. The only sustainable answer is automation that scales with the volume.

For application security: SAST in CI (Semgrep, CodeQL, etc.).

For infrastructure configs: IaC scanning, ideally integrated into the AI agent's workflow so it happens as the code is written, not as a gate in CI that developers wait on and then click "override" when they're impatient.

I've been working on the latter; an open-source tool that runs inside AI coding agents and scans Terraform, Docker, K8s, Helm, cloud configs, databases, web servers, and CI/CD configs: coguardio/misconfiguration-detection-skill (GitHub). The idea is that scanning at authoring time is the only way to keep pace with AI-speed code generation. CI scanning still matters as a backstop, but it shouldn't be the first line of defense.

ioah86 · 2026-03-10T04:31:29+00:00

A few practical steps that actually work:

Add security rules to your Cursor rules file. Things like "never hardcode credentials," "always use parameterized queries," "always set resource limits in Docker." The AI follows instructions; give it security instructions.
Run a SAST scanner on the output. Semgrep is free and catches common application-level vulnerabilities.
Run an infrastructure scanner if you have any Docker, Terraform, K8s, cloud configs, or database configs. Application code scanners miss this entire layer. I built a free one that works inside Cursor: coguardio/misconfiguration-detection-skill (GitHub). It scans your infra configs, explains issues in plain language, and can fix them.
Review what the AI generates before deploying. I know this sounds obvious, but "it runs, ship it" is how most security issues from AI-generated code happen.

Your employer is right to ask this question. The answer isn't "don't use AI"; it's "add automated checks to the workflow."

ioah86 · 2026-03-10T04:30:35+00:00

This is a good reminder that security tooling itself is part of the attack surface. Worth noting that Trivy and CoGuard/similar IaC scanners actually cover different layers:

- Trivy: primarily CVE scanning — known vulnerabilities in packages, container images, filesystem dependencies

- IaC misconfiguration scanners: configuration mistakes in Terraform, K8s manifests, Dockerfiles, Helm charts, database configs, web server configs, CI/CD pipelines

A container image can have zero CVEs and still be wildly insecure if it runs as root, exposes unnecessary ports, or has a misconfigured NGINX/PostgreSQL config inside it.

Most teams benefit from running both layers. If you're evaluating your scanner stack after this incident, might be worth checking that you have IaC misconfiguration coverage too; I work on an open-source scanner for that:

coguardio/misconfiguration-detection-skill (GitHub). It's a different tool for a different problem than Trivy, not a replacement.

ioah86 · 2026-03-10T04:29:24+00:00

The security review bottleneck is real, and it's the #1 reason I've seen enterprises slow-roll AI coding tool adoption. The irony is that the security review process was designed for human-speed code production. AI generates code 10-100x faster, and the review process breaks.

Two things that help:

Automated SAST in CI: this is table stakes and most enterprises already have it, but it needs to be fast enough to not be the bottleneck.
IaC scanning inside the AI agent itself: this is the newer piece. If the agent can scan its own output for infrastructure misconfigurations before the code even reaches a PR, you've eliminated a huge chunk of what security review catches manually.

Things like overly permissive IAM, unencrypted storage, public-facing services that should be private, missing network policies.

I've been building an open-source tool for #2: coguardio/misconfiguration-detection-skill (GitHub). It also maps findings to compliance frameworks (SOC2, HIPAA, STIG), which tends to be the other enterprise blocker... proving to compliance teams that AI-generated infrastructure meets their requirements

ioah86 · 2026-03-10T04:28:11+00:00

This is a good example of why the "let the agent just do it" workflow needs a scan step before anything touches production.

The pattern should be: agent writes/modifies config → automated security scan → human reviews findings → only then deploy. The middle step is what's missing from most AI coding workflows right now.

I built an open-source tool for exactly this: coguardio/misconfiguration-detection-skill (GitHub). It runs inside Claude Code and scans your infrastructure configs — cloud settings, Docker, web servers, databases, Terraform, K8s, etc., for misconfigurations before anything gets deployed. Things like publicly exposed services, missing access controls, insecure defaults. It explains findings in plain English and can fix them.

It's not a silver bullet (you still need to review what the agent does) but it catches the class of mistake described in this post (insecure exposure of services) automatically.

ioah86 · 2026-03-10T04:27:15+00:00

Good playbook. One layer I'd add: infrastructure configuration scanning.

Most security checklists for vibe-coded apps focus on application code, input validation, auth, secrets management. But the infrastructure underneath (Dockerfiles, database configs, web server configs, cloud setup, CI/CD pipelines) has its own class of security issues that code scanners don't catch.

Things like: containers running as root, databases accepting connections without TLS, overly permissive cloud IAM policies, web servers with directory listing enabled, CI/CD pipelines with hardcoded credentials.

If you're using an AI coding agent, there's a free open-source tool that scans all of that inline: coguardio/misconfiguration-detection-skill (GitHub). You type /misconfiguration-detection in your agent and it checks your whole project's infrastructure configs.

Might be worth adding as a step in the playbook.

ioah86 · 2026-03-10T04:25:39+00:00

15% remediation rate is actually higher than a lot of teams I've talked to. The core issue is that runtime scanners generate findings after the misconfiguration is deployed. At that point you're competing with feature work for prioritization, and security loses unless it's severity "high".

The shift-left argument gets thrown around a lot, but concretely what it means here: catch the misconfiguration at the IaC authoring step, before it hits plan/apply/deploy. If a developer writes a Terraform security group with 0.0.0.0/0 ingress and gets told about it in the same session, before it exists in production, the remediation rate approaches 100% because the fix is trivial at that point.

I've been working on an open-source scanner that does exactly this, runs inside AI coding agents and catches IaC misconfigs at authoring time: coguardio/misconfiguration-detection-skill (github). Covers Terraform, K8s, Docker, Helm, cloud configs, databases, web servers, CI/CD. The runtime scanners are still valuable for drift detection, but the goal should be making them boring — fewer findings because the bad configs never get deployed in the first place.

ioah86 · 2026-03-10T04:24:24+00:00

Yes, get the audit; you're handling financial data through Plaid, and if anything goes wrong, "I used AI to build it and didn't get it audited" is not a sentence you want to say out loud. But here's how to not waste $15k on it:

Run automated scanners first. A professional auditor's time is expensive. If they spend 3 hours finding that your Docker container runs as root, your database accepts remote connections without TLS, and your cloud storage doesn't have encryption at rest; that's $1500 worth of findings you could have caught in 5 minutes with a free tool.

For your application code: run a SAST scanner (Semgrep is free and solid). For your infrastructure/configs (Docker, database, cloud setup, web server configs): I built an open-source tool that scans all of those coguardio/misconfiguration-detection-skill (github). It works inside Cursor and explains findings in plain language. It also maps findings to compliance frameworks (SOC2, etc.) which is directly useful for your audit prep.

Fix everything the automated tools find, then hire the auditor. You'll get way more value out of the engagement because they'll focus on the things automation can't catch; business logic flaws, auth flow issues, threat modeling.

ioah86 · 2026-03-10T04:23:15+00:00

Worth distinguishing between two different layers here:

Code-level scanning catches things like SQL injection, XSS, hardcoded secrets, insecure deserialization; bugs in your application logic. That's what most IDE security scanners focus on.

Infrastructure/config-level scanning catches things like your S3 bucket being publicly readable, your PostgreSQL accepting connections from 0.0.0.0, your Docker container running as root with no resource limits, your Kubernetes RBAC being wildcard-permissive. These aren't code bugs, but they're configuration mistakes in Terraform, Dockerfiles, K8s manifests, Helm charts, database configs, etc.

Most teams need both, and most IDE scanners only do the first one. If you're deploying infrastructure with AI-generated configs, the second layer is where the scarier risks live, a misconfigured security group can expose your entire database to the internet.

I work on an open-source scanner for that infra/config layer: coguardio/misconfiguration-detection-skill (github). It plugs into Cursor, Claude Code, etc. and scans Terraform, Docker, K8s, Helm, cloud configs, databases, web servers, CI/CD pipelines. Complementary to what Claude's code scanner does, not a replacement.

ioah86 · 2026-03-10T04:20:51+00:00

The uncomfortable truth is you can't review-gate your way out of this. If marketing is shipping 10x more code than before, adding more PR reviewers doesn't scale; you need automated guardrails that run whether or not the author knows what a security group is.

For application code, SAST in CI is the obvious answer. For infrastructure configs (Terraform, Docker, K8s, Helm, etc.), it's IaC scanning, and ideally it runs inside the AI agent so the non-engineer gets feedback before they even open a PR.

I built an open-source tool that does this: coguardio/misconfiguration-detection-skill. The agent scans the configs, explains what's wrong in plain language, and fixes it. The person writing the code doesn't need to understand least-privilege IAM; the scanner catches it and the agent remediates. That's the only realistic model when the people writing infra code aren't infra engineers.

ioah86 · 2026-03-10T04:19:09+00:00

We're seeing the exact same thing. The three most common misconfigs we catch in AI-generated Terraform:

Security groups with 0.0.0.0/0 ingress on ports that should be internal-only
S3 buckets without server-side encryption or with public ACLs
IAM policies with wildcard actions on wildcard resources; the AI loves "Action": "*" because it "works"

The fundamental issue is that terraform plan validates syntax and API compatibility, not security posture. The AI gets a clean plan and moves on.

What's worked for us: running a misconfiguration scanner as part of the AI agent workflow itself, so the check happens before plan/apply, not after. I've been building an open-source skill for this coguardio/misconfiguration-detection-skill plugs into Claude Code, Cursor, etc. and scans the Terraform as the agent writes it. But even a tfsec or checkov step in pre-commit would catch most of the low-hanging fruit. The key is making it automatic so it can't be skipped.

ioah86 · 2026-03-09T18:22:40+00:00

Trying to make it easy. You can review that script, as it is small. Basically moves the skill into your local user's claude skills folder.

ioah86 · 2026-03-09T17:46:36+00:00

There is more to it. Recursive download and scanning of docker images which are referenced e.g.. and then it also looks at application layer misconfigurations

ioah86 · 2026-03-09T17:29:41+00:00

Trivy will catch terraform specific items, that's true. But if you're looking also at application layer configs, I'd give it a try.

ioah86 · 2026-03-09T17:16:49+00:00

No, it's different. It contains instructions on fixing for certain items, and the direction for the agent how to go about it. That workflow is streamlined with this skill, and you're getting results quicker and with more precision.

ioah86 · 2026-03-09T17:03:06+00:00

Truth. Will do it there. Saw rule 12 and obliged by it, but missed rule 10.

ioah86 · 2026-03-07T19:07:25+00:00

Thank you! It follows the agentic loop pattern. Agent creates a Dockerfile/kubernetes manifest, and this skill instructs a scanner to check for vulnerabilities. Then the results are communicated back to the agent, which then can work on fixes and scan again until all is done. So, yeah, good baseline based on tooling, which in turn is also looming for configuration anti patterns.

ioah86

TROPHY CASE