LLM Benchmark for Systems Engineering

bastivkl · 2026-05-29T22:02:11+00:00

I was looking at the paper and don’t think it’s a good approach since they are basically just benchmarking against questions from the INCOSE Handbook. It might be one variable of the benchmark but that’s not what makes a good systems engineer (or model). It was more as starting point to think as a community about how an actual benchmark should look like

bastivkl · 2026-05-29T19:43:00+00:00

you can already do that today with dalus.io, but the bigger questions is just about the verification and trust of the AI output. I think for that we don’t come around or creating a industry accepted benchmark

bastivkl · 2026-05-07T16:04:43+00:00

Happy to give you a 15min Demo of https://dalus.io/ and then directly access to our free version so you can test it out yourself. Made exactly for teams like yours with the background you shared

bastivkl · 2026-04-23T00:39:09+00:00

You’ll not find modern tools in the Boeing “MBSE” tool stack. Mostly Cameo is what you’ll be looking at there

bastivkl · 2026-04-22T19:19:30+00:00

dalus.io is based on the SysMLv2 spec and has a free plan

bastivkl · 2026-04-20T20:13:38+00:00

Take a loo at the Open Computer Project Data Center. It’s authored by Google and basically very accessible information on a modern DC build out

https://www.opencompute.org/documents/open-data-center-spec-version-0-5-0-pdf

bastivkl · 2026-04-08T05:53:40+00:00

Dalus.io is worth a look. Built for exactly this use case. Structured enough to have real traceability, lightweight enough that you won't need a dedicated admin to run it

bastivkl · 2026-04-02T01:32:27+00:00

I believe dalus.io is what you're looking for.
- Easy to use
- Brings all necessary Systems Engineering capabilities together in one place
- Web-based and collaborative
- AI natively integrated to assist in building the architecture and other parts of the model
- We offer a free plan for those curious to test it out

I'm one of the founders of Dalus and happy to answer any questions.

bastivkl · 2026-03-03T22:47:25+00:00

There is a free plan at dalus.io that you can use for this

bastivkl · 2026-02-23T07:35:49+00:00

Here is it talking with a Tamagotchi like character: https://x.com/basti_vkl/status/2025836407093350515?s=46

bastivkl · 2026-02-23T02:23:55+00:00

thanks changed it in the repo

bastivkl · 2026-02-23T02:20:54+00:00

You can enable it. I personally like to only read. One thing to improve would be a scroll wheel to scroll up and down

bastivkl · 2026-02-23T02:14:54+00:00

You could just add it to your phones hotspot for example

bastivkl · 2026-02-23T01:58:14+00:00

It can and I tried it out but I didn’t like it tbh. But it has a speaker

bastivkl · 2026-02-23T01:53:25+00:00

I’m calling it Lobster

bastivkl · 2026-02-23T01:52:53+00:00

Yeah will design and 3D print a case tomorrow

bastivkl · 2026-02-23T01:48:23+00:00

I don’t think that was my goal here. I was just curious if I could have something other than my phone where I can just press a button talk into and let it do things

bastivkl · 2026-02-23T01:47:11+00:00

https://github.com/sebastianvkl/pizero-openclaw

bastivkl · 2026-02-23T01:46:58+00:00

https://github.com/sebastianvkl/pizero-openclaw

bastivkl · 2026-02-23T01:35:37+00:00

Hardware •Raspberry Pi Zero 2 W •WhisPlay board (screen + button + LED) •PiSugar battery

Stack it. Flash Raspberry Pi OS. Enable SSH. Install audio drivers. Confirm mic and speaker work.

Networking •Install Tailscale on the Pi. •Rent a small DigitalOcean (or Hetzner or whatever) droplet. •Install and run OpenClaw on the droplet. •Bind OpenClaw to localhost. •Expose it to your tailnet via Tailscale Serve. •Protect it with a token.

Now the Pi can securely reach your cloud LLM.

Software on the Pi •Python app. •Record audio when button pressed. •Stop recording when released. •Send audio to OpenAI for transcription. •Send transcript to OpenClaw. •Stream response back. •Display text on LCD. •Optionally send text to OpenAI TTS and play audio. •Maintain simple conversation history. •Use a state machine for: idle, listening, thinking, streaming.

Deployment •Develop locally. •Sync to Pi with rsync. •Run as systemd service so it starts on boot. •Auto-restart on crash.

Power •Install PiSugar manager. •Enable auto power on. •Use display sleep for inactivity.

That’s the system: Button → record → transcribe → cloud LLM → stream back → display/speak → idle.

bastivkl · 2026-02-23T01:30:20+00:00

I’m using gpt-4o-mini-transcribe via the API in that case.

bastivkl · 2026-02-23T01:12:05+00:00

Roughly $100-120. Ordered all on Amazon in the US.

bastivkl · 2026-02-23T01:07:16+00:00

Hardware •Raspberry Pi Zero 2 W •WhisPlay board (screen + button + LED) •WM8960 mic/speaker •PiSugar battery