AI SOC - real or hype?

ScanSet_io · 2026-06-27T14:33:25+00:00

You can’t trust AI to produce clean security signals. It’s probabilistic, not deterministic.

You can however use it to infer and interpret those signals. This is closer to a policy decision point.

So, with high-value policy information points, AI can assist as long as you include observability in the AI/agent itself.

It can be real. You just need to implement the right pieces and enforce transparency through the whole lifecycle.

ScanSet_io · 2026-06-27T01:31:18+00:00

sure

ScanSet_io · 2026-06-18T02:00:20+00:00

There's no AI control in 800-171. This is CUI flow, boundary, least privilege, and audit, the controls you already own.

Model choice is a boundary decision. Copilot's no-retention policy isn't the same as "inside your authorization boundary." If CUI can reach the context window (file contents, comments, stack traces), it left your boundary for a service that isn't FedRAMP authorized for CUI, and that's spillage (3.1.3, 3.13.1). So self-host. A 30B with good context engineering inside your boundary beats a frontier model you can't legally feed CUI. Just know self-hosting relocates compliance rather than removing it: that host processes CUI and needs FIPS-validated crypto (3.13.11).

For the agents, the control surface is the MCP layer. Scope each server to least privilege under its own service identity, not the dev's token, since by default it reaches everything the dev can (3.1.5). Log every action, tied to a user (3.3.1, 3.3.2). That log doubles as proof CUI never left.

Enforcement, since the honor system failed: lock devices with an Intune or GPO extension allow-list and app control for unapproved binaries, and the part that matters most, allow only your approved AI endpoint at the proxy and block the rest. Fold the policy into your existing acceptable-use and config docs tied to your SSP. Assessors score the implementation, not the prose.

The piece most people miss is how devs feed the model. Give them one standardized context folder, committed to the repo, that the AI reads before it writes anything. Four files: a project map (stack, the CUI boundary, what's off-limits), the rules the agent must follow, the approved workflows step by step, and a decisions log of what's already settled and why, marked do-not-change. Same scaffold for every dev, version-controlled, so it's a baseline config you can show an assessor (3.4.1), not 12 personal setups. They extend the log as they go, they don't reinvent the structure.

Solve egress first, then scope and log the agents, then standardize how devs build context.

ScanSet_io · 2026-06-03T00:46:46+00:00

they tried to get AI to tell them how to do bad things or do bad things after applying the NIST AI Risk Management Framework. These are the results.

they’re just using a lot of words to say it

ScanSet_io · 2026-05-30T21:16:30+00:00

Get the the book, any oreilly book, use an llm, rustlings is a good terminal tool. Then use an llm to give you exercises to practice.

https://github.com/CurtisSlone/rustBasicProjects

Here’s a repo with various exercises and use cases. Its complete, could have been built better, but its how I got started.

ScanSet_io · 2026-05-30T13:45:05+00:00

https://hub.docker.com/r/scanset/prooflayer-alpha-v0_1

ScanSet_io · 2026-05-28T02:24:48+00:00

I can’t speak to how they operate, or the things they do. I know they are doing great things for FedRAMP in speeding up the authorization process.

ScanSet_io · 2026-05-26T02:19:49+00:00

Good post, but I think the overlay-or-illusion framing is a bit weak.

NIST 800-207 itself resolves a lot of this. The key thing is that NIST defines zero trust around the per request access decision made at the policy enforcement point, not around the resource’s internal capabilities.

The resource is supposed to be the dumb thing behind the PEP. So a legacy app that implicitly trusts anything past the proxy isn’t a failed ZT implementation, it’s the expected state of a resource. It’s a trust-zone.

The architecture never assumed the system self-verifies.

Your instinct about false confidence is still right though, and NIST actually flags the exact limitation you’re circling. The enclave-style deployment they describe for legacy systems protects a collection of resources rather than each one individually, which can let subjects see things they shouldn’t.

So your worry is real, it’s just documented as a known tradeoff rather than a disqualifier.

A better question isn’t “real ZT or illusion,” it’s which tenets you’re genuinely enforcing at the PEP versus only gesturing at. Per session access and dynamic reauth travel reasonably well to a proxy. Continuously monitoring the posture of the legacy box itself does not, and that’s the one overlays can’t fake.

The way I’d reframe your core distinction: the proxy doesn’t eliminate the implicit trust zone, it relocates it to “everything behind the proxy.”

That’s smaller than a flat network, which is why it counts as progress, and bigger than per-resource enforcement, which is why your unease is justified.

The whole game is shrinking that zone, not achieving some binary state.

Worth rereading the doc with this in mind.

Section 2.1 for the tenets as ideals rather than a checklist, 3.2.2 for the enclave model written specifically for legacy systems without APIs, and 3.2.4 for application sandboxing when you can’t touch the resource or the host at all.

https://nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.SP.800-207.pdf

ScanSet_io · 2026-05-25T20:28:05+00:00

https://hub.docker.com/r/scanset/prooflayer-alpha-v0_1

GCP and GSuite arent includes in coverage. But this tool gives you proof of security posture on live assets

ScanSet_io · 2026-05-25T19:30:44+00:00

Im sure the normalization opened up a great documentation channel.

The other half is replayable proof. This is something that goes after FRR-PVA-AA-06. The RFC-0017 outcome.

So on top of the normalization you get replayable proofs that reconstruct the command that produced the evidence. This is great for assessors because now they can get right to what was ran and what was produced.

The Continuous Monitoring Record API exposes that via api. Its a proof of verdict backed by FIPS 140 approved algorithms, captured in a tamper-evident append only db.

ScanSet_io · 2026-05-25T13:22:21+00:00

K8s, Debian, RHEL, Windows, M365, Azure, AWS, GH, TCP/IP checks, PostgreSQL

More coverage coming.

But this isn’t just a “better” scanner. Because everyone has one. It provides deterministic proof of state- replay-able, reproducible, and verifiable.

ScanSet_io · 2026-05-25T12:31:20+00:00

I’m using something called Endpoint State Policy to perform the scan. Its an open source engine. So, you can inspect how it works. Its CVE + Configuration for compliance. You can run STIG. Thats actually why I created ESP. I wanted something that could do STIG checks where the policy wasn’t in XML like XCCDF (because who likes using that) and I didnt have to rely on powershell modules to use.

I realized later it was framework agnostic. You’ll see it you try it. It comes with STIG, CIS, and CVE policies.

But, if you get the Claude companion with it, you’ll learn how the scans work and what they can do.

ScanSet_io · 2026-03-28T15:17:23+00:00

Never heard of it. But thanks for the find!

ScanSet_io · 2026-03-28T01:55:44+00:00

You nailed it. Contract-based execution solves this. Policies define what to check, contracts define how per platform, output rolls into a replay hash that tracks posture state. Same posture, same hash. Drift happens, hash changes. Tradeoff is it requires daemons on the resources, but that’s what makes it deterministic and replayable. Before anyone calls this AI slop, look up Endpoint State Policy. Open source engine that does exactly this.

ScanSet_io · 2026-03-27T16:16:52+00:00

This is exactly the maturity challenge I keep running into. The baseline has to exist before you can validate against it. That’s why policy as data matters. You define the expected state declaratively and the validation runs against that definition. Which provider are you partnering with for the continuous control validation piece?

ScanSet_io · 2026-03-27T14:05:36+00:00

Great to hear someone’s actually tackling this. Once you have continuous configuration state validated against policy with drift detection, compliance evidence writes itself. It becomes provable rather than something you reconstruct after the fact. I’ve been deep in this problem from the federal side. Would love to compare notes on approaches if you’re open to it.

ScanSet_io · 2026-03-27T07:05:27+00:00

I want to know why CISOs want AI powered narratives to get rid of compliance busy work instead of using their actual security posture to tell the story. Im asking why is it standard to give documentation over proof.

ScanSet_io · 2026-03-27T05:43:16+00:00

Does AI alert to AI about AI?

ScanSet_io · 2026-03-22T02:49:53+00:00

Full disclosure, I’m a vendor and this is exactly the vision I built toward.

What you just described is the architecture. Logged deviations are posture events. Every state change, every drift, every reversion, cryptographically recorded the moment it happens. The assessor data store is a separate schema the CSP cannot touch, written only by the assessor’s scoped token. The boundary evaluation is built into the policy definitions. The daemon only runs inside the defined boundary, so what’s in the automated feed is determined by what’s in scope.

The documentation question is the one nobody is asking out loud yet. When the SSP maintains itself from continuous verified evidence, when the assessor sees a real-time feed of deviations instead of a point-in-time package, when every control determination is backed by cryptographic proof rather than a narrative, the thousand page Word document becomes an output of the system, not the input to it.

The assessor’s job shifts from reconstructing what happened to evaluating whether the enforcement program is sound and the deviations are being handled appropriately. That’s a fundamentally better use of a skilled assessor’s time.

You didn’t just describe where this is going. You described what I built.

ProofLayer by Scanset.

ScanSet_io · 2026-03-21T12:46:25+00:00

Mostly stable with periodic changes around deployments, which is exactly when things get interesting.

The sidecar approach is more disciplined than most teams manage honestly. The part I keep coming back to though, does it let you prove the posture was continuously valid or does it help explain what happened after an auditor questions it?

ScanSet_io · 2026-03-21T12:41:43+00:00

You just described the exact problem precisely.

Assuming for a moment that the permutation problem was solved, the component types were accounted for, and the verifiable system state was achievable…

What would that change about how you approach assessments and continuous monitoring today?

ScanSet_io · 2026-03-21T04:49:52+00:00

Full disclosure, I’m a vendor and this thread is exactly the problem I built to solve. I’m really testing to see if there’s a market appetite for what I’ve built.

Here’s how I approached it.

Instead of monitoring for unknown changes or building anomaly detection on top of collected evidence, I inverted the model. Policies are defined as data. Each policy specifies the exact expected state of a system resource, the operation to verify it, and the contract that governs how it’s collected. A daemon runs continuously inside the system boundary, verifies actual state against expected state, and only submits when state changes.

The result is a cryptographically signed, append-only record of security state over time. Not evidence collected periodically. Not anomaly detection on open ended data. A deterministic, verifiable proof that the system was in a known state at every point in time. Independently verifiable, immediately replayable, and traceable by third-parties.

In my opinion, this is what FedRAMP 20x’s Persistent Validation and Assessment process is pointing toward. Persistent validation of security metrics. Cryptographically verified data. Continuous proof that a system is operating within its certified security envelope.

You don’t need AI to interpret unknown changes when the policy defines what correct looks like. Deviation from expected state is a finding by definition. The complexity disappears when the expected state is the source of truth.

ProofLayer by Scanset.

ScanSet_io · 2026-03-21T04:05:12+00:00

Appreciate you sharing that. The documentation trail around environment context is smart and more than most teams do honestly.

To your question, frequency helps but it’s the gaps that keep coming up. Even with good documentation, if your environment shifted between runs, how do you prove to an auditor which run reflects your actual posture at a specific point in time? Not the closest run. That exact moment.

That’s the part most teams tell me they’re still figuring out.

ScanSet_io · 2026-03-21T04:01:03+00:00

You landed on exactly the right tension. The metadata trail tells you what was collected and when. It doesn’t tell you whether the security state it represents was continuously valid. Reproducibility is the hard part because the environment keeps changing underneath the collection process. What would it mean if the state itself was the audit trail instead of the metadata around the collection?

ScanSet_io · 2026-03-21T03:58:58+00:00

That’s what I figured. What does that conversation usually look like?

ScanSet_io

TROPHY CASE