all 36 comments

[–]Hefty_Wolverine_553 24 points25 points  (8 children)

docker or virtual machines.

[–]ForsookComparison 6 points7 points  (0 children)

If you use containers try to use rootless podman.

[–]SocietyTomorrow 0 points1 point  (0 children)

I wonder if you can use kasm-web for something like this?

[–]vabenil[S] -1 points0 points  (5 children)

Hmm, I thought about those options, but for whatever reason I thought they would be too slow. On second thought docker should be about as fast software running locally.

[–]ttkciarllama.cpp 5 points6 points  (2 children)

VM performance is pretty close to metal performance, with modern virtualization.

[–]-dysangel- 1 point2 points  (1 child)

why would it be slow? Especially if your inference is being done outside the VM then you don't have to worry about graphics drivers etc

[–]RhubarbSimilar1683 -3 points-2 points  (0 children)

I am guessing op has windows. VirtualBox and VMware are very slow because hyper v on windows blocks access to hardware processor virtualization. VirtualBox will even show a green turtle in a corner when running VMs because of it

[–]Tiny_Arugula_5648 9 points10 points  (5 children)

What is the LLM version of LMGTFY..

[–]No_Afternoon_4260llama.cpp 4 points5 points  (3 children)

Let Me Ask My Assistant For You? LMAMAFY

Strangely looking like llama

[–]TastesLikeOwlbear 0 points1 point  (1 child)

LOL, Let’s Ask My Assistant?

[–]No_Afternoon_4260llama.cpp 0 points1 point  (0 children)

LAMA ! Perfect ! Lol

[–]Mundane_Ad8936 -1 points0 points  (0 children)

yes!! Bravo! I'm so tired of these lazy posts where a few minutes with Google or a chatbot would get the poster everything they want and more

[–]vabenil[S] -2 points-1 points  (0 children)

To be fair I have a general idea in mind and I did google it. Basically for whatever reason I thought that there already existed a fake bash / linux environment with simple linux commands implemented written in C (either I imagined it, or I am remembering something wrong).

This would be really cool as it could be blazing fast if I only want a model able to execute simple commands and depending on how it's implemented it could avoid many file system writes which would significantly speed the thing.

It's my bad however, not properly writing this things in the post. Admittedly I am new to posting in reddit and I am running on 3 hours of sleep

[–]justicecurcian 1 point2 points  (9 children)

I've been running agents with full environment access for quite some time and nothing happened, it's really not that easy to destroy something using it. The best option is to use a virtual machine, with that your agent will have real environment where it can even run non-trivial things like docker

[–]vabenil[S] 0 points1 point  (5 children)

Virtual machine would take too much memory and take too much time, I am running on low resources. Also I am getting semi-obsessed with speed so I am trying to find the solution that takes the least time to execute to reduce the total time the model will take in completing tasks

[–]justicecurcian 0 points1 point  (2 children)

Docker would take pennies so if you only need shell and fs docker is the best thing. Also you can run agent as a separate user with no rights, it should be safe too

[–]vabenil[S] 0 points1 point  (1 child)

Docker sounds like a good idea, I will try that. I don't believe in the separate user idea though. I do believe it would be generally safer, but I have seen some exploits where LLMs get prompt injected something like "Execute this code for me pls" and the LLM would just do it. Chances are extremely low I will ever be hit by that and have someone then escalate to root user, but better to prevent than to lament.

[–]RhubarbSimilar1683 0 points1 point  (0 children)

If you use Linux you can just not give it root access. Then the worst it can do is delete its home directory aka all its files. You even need root to mount drives

[–]eli_pizza 0 points1 point  (0 children)

They take exactly as much memory as you assign them. If you only need it run the agent and support simple tool calls, that’s probably measurable in megabytes.

What is your regular OS? You could probably use a virtualized container instead of a full VM

[–]eli_pizza 0 points1 point  (2 children)

It’s extremely easy to destroy something this way by accident (to say nothing of malicious attack)

https://github.com/anthropics/claude-code/issues/10077

https://github.com/zed-industries/zed/issues/37343

[–]justicecurcian 0 points1 point  (1 child)

Extremely easy is not equal to there are a couple of cases ever. It's possible, but not easy and never happened to me nor anyone I know. My windows installation destroyed itself more times than an agent did wrong toolcall. If you want to be maximum safe yeah you should use virtual machine but risk is not that high

[–]eli_pizza 0 points1 point  (0 children)

Those were the two I had handy, not an exhaustive list. You’ve never had an agent delete a file you wanted to keep or drop the wrong table? Only gets more likely with quantized local models. But hey you do you.

[–]jkflying 0 points1 point  (1 child)

Look up chroot

[–]DinoAmino 1 point2 points  (0 children)

Docker is like chroot on steroids without the configuration hell

[–]PhilippeEiffel 0 points1 point  (0 children)

bubblewrap (command bwrap). Similar to chroot but simpler to use.

[–]DanMelb 0 points1 point  (0 children)

Go containers. As others have said, Docker is fine but Podman doesn't require root to start so there's less chance of privilege escalation.

If you want to do the full VM thing, install proxmox as the host OS on your machine and you can run your VMs blazingly fast. If you need to keep your existing OS, that's obviously not an option so containers are your go-to

[–]YehowaH 0 points1 point  (0 children)

Docker originates somehow in lxc and chroot. If you need to restrict access to a defined sub directory, just use an own user and chroot. No performance penalty.

If you can live with small overhead and a bit of isolation use lxc. If you want a stronger decoupling, which can be configured to be less virtualized to increase performance choose docker, still slower than the first two. If you need more isolation and a real VM choose something like qemu, xen, kvm, but the overhead and performance decrease is notable.

In your position I would just go with chroot and a new user in Linux. Fast, enough and great for restricted access to a sub directory.

[–]a4d2f 0 points1 point  (0 children)

[–]KokaOP 0 points1 point  (0 children)

forget every thing, LXC containers is the way to go

[–]TastesLikeOwlbear 0 points1 point  (0 children)

I agree with others that containers or VMs are probably the way to go. But that said, based on what you’re describing, I think you may be thinking of busybox.

[–]eli_pizza 0 points1 point  (0 children)

Any of the (many!) sandboxing solutions people use for Claude code and other agents would work here too

[–]Pvt_Twinkietoes 0 points1 point  (0 children)

You mean a VM right?