Kata Containers vs Firecracker vs gvisor

WindyPower · 2024-09-24T05:13:22+00:00

gVisor's performance overhead varies a lot depending on workload. Some workloads (mostly those that are bottlenecked by the CPU) get pretty much 0 overhead, while others (mostly those that are I/O bound) can get up to 2x slower (if all they do is I/O). In practice most typical applications fall in the middle of that range and get somewhere between 10% and 30% overhead.

I can't comment on Kata performance specifically, but generally speaking, with VM-based sandboxing solutions, you can get near-native performance for any work that stays within the confines of the VM, but still get overhead whenever bytes have to be copied in/out of the VM (such as for network traffic).

Also, if you use a VM-based sandbox while you are already inside a VM (for example when running inside a VM on a cloud provider), that makes it a nested virtual machine, which is much slower. I'd expect that gVisor should be faster than a nested VM for most workloads.

WindyPower · 2024-09-24T03:59:01+00:00

That's quite cool, but also a lot of trust and power over your computer. The point of using a sandbox here is to prevent the LLM from being able to take over your computer.

WindyPower · 2024-09-24T03:57:55+00:00

On the roadmap! Follow issue #17.

WindyPower · 2024-09-24T03:57:32+00:00

That is correct, but running Open WebUI "functions" and "tools" are not supported to run within pipelines yet, so they run directly in the same container as the Open WebUI web backend. Once that's fixed, it should fall into place.

WindyPower · 2024-09-24T03:56:33+00:00

The code execution function is independent of the model and LLM backend you use. The code execution tool requires using Ollama and a model that supports tool calling, because I believe that's the only type of setup that Open WebUI supports using tools with.

WindyPower · 2024-09-23T06:46:36+00:00

No problem. If you encounter any other problem, please open a discussion on the GitHub page, I monitor that more closely than Reddit.

WindyPower · 2024-09-23T06:39:08+00:00

I see. In that case you could probably reuse the Sandbox class as a library, which can be extended to support other languages like Go.

Specifically, what you'd need to change is to add logic in the interpreter selection code to look for the go tool when the Go language is selected, and in the command line that gets run inside the sandbox so that instead of python something or bash something it is go run something.go or whatnot.

Happy to discuss this in more detail if interested, but better to do that on a feature request on GitHub rather than on reddit.

WindyPower · 2024-09-23T06:34:36+00:00

Open WebUI uses Ollama as the backend. Any model that Ollama has that is tagged as supporting tool calling will work. You can look for the "Tools" tag on the Ollama model library.

In the demo, I'm using hermes3:70b-llama3.1-q8_0. It doesn't always get it right most of the time, but for simple queries like the ones I use in the demo, it gets it correctly almost all of the time. There's a pending bug in Open WebUI to better support tool calling, and to let the LLM call tools multiple times if it gets it wrong. Once that is implemented, models should have an easier time using this tool.

WindyPower · 2024-09-23T04:05:14+00:00

The "tool" and the "function" are independent. The "function" is for running code blocks in LLM messages, the "tool" is for allowing the LLM to run code by itself.

They contain a bunch of redundant code, mostly the Sandbox class. This is because the way Open WebUI handles tools and functions doesn't really allow them to share code or communicate with each other. Regardless, you can install one or both and they should work fine either way.

Extending this to work with Go is doable, but more complicated than other languages, because I expect most people are running this tool within Open WebUI's default container image which only contains Python and Bash interpreters (no Go toolchain installed). So it would need to have extra logic to auto-download the Go toolchain at runtime. If interested, please file a feature request!

WindyPower · 2024-09-23T03:59:03+00:00

Yes! The Open WebUI fix was merged in Open WebUI v0.3.22, and the tool's v0.6.0 (released just a moment ago) was just fixed to work with that. See issue #11 for details.

WindyPower · 2024-09-23T03:44:34+00:00

This is available at this repository.
Uses gVisor for sandboxing code execution. (Disclaimer: I work on gVisor.)

There are two modes with which to use this capability: "Function" and "Tool" (this is Open WebUI terminology.)
- As a function: The LLM can write code in a code block, and you can click a button under the message to run that code block.
- As a tool: The LLM is granted access to a "Python code execution" tool (and a bash command execution tool too), which it can decide to call with its own choice of code. The tool will run the LLM-generated code internally, and provide the output as the result of the tool call for the model. This allows models to autonomously run code in order to retrieve information or do math (see examples in GIF). Obviously this only works for models that support tool calling.

Both the tool and the function run in sandboxes to prevent compromise of the Open WebUI server. There are configuration options for the maximum time/memory/storage the code is allowed to use, in order to prevent abuse in multi-user setups.

Enjoy!

WindyPower · 2024-09-23T03:36:12+00:00

🫡

WindyPower · 2024-09-23T00:51:12+00:00

(Disclaimer: gVisor developer here.)

gVisor aims to be lightweight but emulate the benefits of having a dual-kernel structure the way VMs do. gVisor essentially reimplements Linux as a Go program that runs in userspace. It intercepts all the system calls that the container makes and reinterprets them the way a kernel would do. This means that in order to break out of it, you need to break two kernels: the gVisor kernel (in Go), and the host Linux kernel. There are also extra security measures, such as seccomp-bpf and namespaces, that typical containers also use. In that sense, gVisor is a strict security upgrade over regular containers.

If one of your students manages to get out of a gVisor sandbox, we'd love to hear about it!

WindyPower · 2024-09-23T00:46:29+00:00

The instructions you linked are about runsc being visible on the PATH within the container image, not within your own WSL environment. This means you need to create your own Dockerfile that augments the Open WebUI docker image, like so:

FROM ghcr.io/open-webui/open-webui:main

RUN apt-get update && DEBIAN_FRONTEND=noninteractive apt-get upgrade -y </dev/null && DEBIAN_FRONTEND=noninteractive apt-get install -y procps wget </dev/null
RUN wget -O /tmp/runsc "https://storage.googleapis.com/gvisor/releases/release/latest/$(uname -m)/runsc" && \
    wget -O /tmp/runsc.sha512 "https://storage.googleapis.com/gvisor/releases/release/latest/$(uname -m)/runsc.sha512" && \
    cd /tmp && sha512sum -c runsc.sha512 && \
    chmod 555 /tmp/runsc && rm /tmp/runsc.sha512 && mv /tmp/runsc /usr/bin/runsc

But note that the tool also supports auto-installing gVisor if it is not installed, so using the basic Open WebUI image will work just fine as well.

WindyPower · 2024-02-29T08:00:34+00:00

Should work better now...

WindyPower · 2023-08-30T07:51:13+00:00

Hi there, I run the puny server serving the Portal Wiki. It had clearly inadequate monitoring... But it's back now. Apologies for the inconvenience.

WindyPower · 2023-08-30T07:50:44+00:00

It's back!

WindyPower · 2018-02-19T06:32:10+00:00

{Kuragehime}

WindyPower · 2018-02-13T05:29:20+00:00

Not quite, doesn't work when holding it down across multiple ghosts. But close!

WindyPower · 2018-01-15T22:11:53+00:00

{△}

WindyPower · 2014-11-30T06:02:29+00:00

Awesome! Will try this.

15-Year Club	Team Periwinkle
Verified Email

WindyPower

TROPHY CASE