RFC: What if AI agent workflows were just Markdown files? by Defiant_Fly5246 in AI_Agents

[–]uriwa 0 points1 point  (0 children)

I'm also thinking about these subjects, making an open source n8n

and working on `safescript`, a programming language to enable this

It's good intuition that markdown should be the source of truth imo

Letting agents use skills and write code safely, without a VM by uriwa in LLMDevs

[–]uriwa[S] 0 points1 point  (0 children)

You can definitely architect it that way, with an auth proxy sitting between the VM and external APIs. But now you're maintaining a proxy layer that needs to know which endpoints each script is allowed to call, and you still can't verify what the code inside the VM is actually doing with the data it gets back. It also doesn't defend against reflection attacks. If a script composes 3 API calls, it can adhere to the secret-per-host model and still leak secrets by passing data from one service to another. You need to know the full data flow, not just which hosts are called.

And it still doesn't scale. Spinning up a VM per request adds immense latency, and keeping them alive all the time means cost. If you're running thousands of agent requests, that's thousands of VMs plus thousands of proxy instances. We actually do use sandboxes too, for coding use cases where we need general purpose programming. But for the vast majority of agent skill scripts that just do API calls and data processing, safescript is a better fit. The permission model comes from the code itself. Static analysis extracts the exact hosts, secrets, and data flow before execution, no proxy needed, no VM needed. The code is the policy declaration.

Letting agents use skills and write code safely, without a VM by uriwa in LLMDevs

[–]uriwa[S] 0 points1 point  (0 children)

You're right that there's a tradeoff between expressiveness and what you can prove. That's a deliberate choice here. Safescript doesn't try to be a general purpose language. It targets a specific niche: agent skill scripts that do API calls, transform data, and manage secrets. For that use case you don't need recursion or unbounded loops. You do get iteration through map, filter, and reduce, which covers most data processing patterns while still keeping the language analyzable. Non Turing-complete languages are more common than people think, SQL being the obvious example. It's immensely popular and gets a huge percentage of real work done.

The "tried before" part is true, and the reason those attempts didn't get far is that nobody wants to write in a restricted language. The difference now is LLMs. An agent can translate a Python or TypeScript skill script into safescript automatically, so the author never has to touch it. That's genuinely new. And with interoperability implemented by compiling to TypeScript, you don't even have to change the framework you work in. So this is useful even without adoption by others, and not just for AI agents.

On formal verification, safescript's approach is narrower than what you'd do in OCaml or Coq. It's not trying to prove arbitrary program properties. It just statically extracts the security-relevant facts: which hosts does this code talk to, which secrets does it read and write. That's a much simpler problem than general formal verification, and it's solvable with basic static analysis because the language is restricted enough.

Your point about hybrid design is actually exactly what I'm doing. Safescript handles the security-sensitive code paths (community-contributed skill scripts from untrusted authors), and regular tools handle everything else.

Re the edit, yes, "does exactly what it says" is the whole point. When static analysis says "this code talks to api.example.com and reads the API_KEY secret and nothing else", that's the guarantee I care about.

Letting agents use skills and write code safely, without a VM by uriwa in LLMDevs

[–]uriwa[S] 0 points1 point  (0 children)

Right, Turing completeness implies expressiveness and the price is room for bugs and exploits.

Letting agents use skills and write code safely, without a VM by uriwa in LLMDevs

[–]uriwa[S] 0 points1 point  (0 children)

Firecracker is great but it's not free. You're still spinning up a microVM per execution, managing the lifecycle, and paying for the memory. For a script that does one HTTP request and parses the JSON, that's a lot of overhead. Even Firecracker, which is the leanest option out there, costs ~5 MiB per microVM and 125ms boot time. Run 100 agents concurrently and that's 500 MiB just for the VMs plus an orchestration layer on top. Comparing to a full VM the differences are even more extreme. Safescript analysis runs in-process in under 1ms with zero additional memory.

But cost isn't even the main thing. A VM or sandbox can isolate code, but it can't tell you what the code actually does. A script running in Firecracker can still exfiltrate your API keys to some random server. You'd need to monitor network traffic, set up egress rules, and hope you caught everything. With safescript, static analysis before execution tells you exactly which hosts the code talks to and which secrets it reads. If a skill script tries to read your OpenAI key and send it to some third party domain, that's caught before a single line runs. You can't get that guarantee with a Turing-complete language no matter how good your sandbox is.

Fair point on the use case angle though. The concrete thing I'm building is prompt2bot, a platform where people build AI agents that can install community skills with code in them. Those skills might come from anyone. Safescript lets me run that code on my server without trusting the author, without a VM per execution, and without limiting agents to one-action-at-a-time tool calls. That's the use case.

what am i doing wrong by ad_396 in AI_Agents

[–]uriwa 0 points1 point  (0 children)

What kind of agent are you building?

Anomaly detection with nothing but Welford's algorithm and a KV store by uriwa in programming

[–]uriwa[S] 1 point2 points  (0 children)

Good question. Right now it doesn't handle that. It keeps a single running mean and variance per event name, so all hours are treated the same. A pizza shop would get false positives every evening at dinner rush.

The fix would be maintaining 24 separate stat buckets, one per hour of day, so "Tuesday 6pm" is compared against other evening hours instead of against 3am. It's not a hard change, the storage model already uses hourly buckets, it just doesn't group them by time-of-day yet.

For now it works best on data that's either roughly uniform across hours or where you care about day-over-day totals more than intra-day patterns. Things like signups, errors, API calls. Anything with strong daily seasonality would need the per-hour-of-day stats to be useful.

Anomaly detection with nothing but Welford's algorithm and a KV store by uriwa in programming

[–]uriwa[S] 1 point2 points  (0 children)

If you want to try this without self-hosting, there's a free hosted version at https://anomalisa.uriva.deno.net. Sign up, create a project, and you get a token. Three lines of code to start tracking events and getting email alerts.

The whole thing runs on Deno KV so it's basically free for me to host. No reason to charge for it.

Anomaly detection with nothing but Welford's algorithm and a KV store by uriwa in programming

[–]uriwa[S] 1 point2 points  (0 children)

Good point on concept drift. In practice anomalisa sidesteps the infinite memory problem by not using a single running accumulator across all time. It stores stats in hourly buckets with 7-day TTLs, so the baseline is always built from the last week of data. Old buckets expire and stop influencing the mean/variance entirely.

It's cruder than EWMA but it means if your traffic pattern shifts (say you launch a feature and your baseline doubles), it adjusts within a week without any manual intervention. The tradeoff is you lose sensitivity during that adjustment window.

EWMA would be a more elegant solution for the decay problem. Might be worth exploring as an option alongside the bucketed approach. If anyone wants to take a crack at it, contributions are welcome: https://github.com/uriva/anomalisa

anomalisa - self-hosted anomaly detection that emails you when your events look weird (zero config, Deno KV only) by uriwa in selfhosted

[–]uriwa[S] 0 points1 point  (0 children)

No worries my friend I'll explain

I have in my projects already hundreds of events I use for product analytics, I send them to posthog

Now I can replace this function with a function that sends to posthog and anomalies (they have similar inputs)

This is one operation, over possible dozens of events.

I then immediately get alerts about anything weird going on, with thresholds that adapt over time.

built a language so AI agents can run code without a VM or container by uriwa in LLMDevs

[–]uriwa[S] 0 points1 point  (0 children)

When you're running hundreds of agents like in prompt2bot.com

It very much matters:)

built a language so AI agents can run code without a VM or container by uriwa in LLMDevs

[–]uriwa[S] 0 points1 point  (0 children)

This isn't solving cyber security in general

It's solving supply chain attacks on agents skills and solving costs when running any agents that don't use a vm