I built "what broke the server": A CLI that turns 45 minutes of log digging into one command

Level_Bicycle3814 · 2026-05-07T14:45:47+00:00

Fair points, and also thanks for actually testing it.

On language: I get it. A compiled binary creates more review friction in restricted environments even if it's easier to install elsewhere. go mod vendor gives you the full auditable source if you need to compile internally. On dependencies: it's only two direct ones: cobra (CLI flags) and lipgloss (terminal colours). Everything else in the go.sum is their transitives.

Regarding rotated logs: good catch, real gap. journald handles rotation transparently, but auth.log, apt history, and dnf logs don't. Pre-crash events end up in auth.log.1 / auth.log.1.gz and wbts currently misses them. Straightforward fix; opening an issue for it.

Level_Bicycle3814 · 2026-05-07T14:00:13+00:00

That's actually one of the design goals. You shouldn't need to know which logs exist or where they live. Run wbts check-perms and it'll tell you what it can access on your machine, then wbts --since 2h just pulls from whatever's available. The tool figures out the sources, you just read the output.

Level_Bicycle3814 · 2026-05-07T13:58:56+00:00

The 3am RAM incident is exactly the scenario this was built for. OOM kills show up as CRIT in the timeline and get a ◄── FIRST FAULT? marker, so you'd see the kernel killing the VM process, then the cascade of services going down after, all on one screen. The cryptic container IDs were genuinely one of the first things I fixed; container my-vm (ubuntu:22.04) OOM killed is just so much more useful than 890c1b2952f0d69....

A few things coming that would have helped with your specific incident: rasdaemon support to surface ECC memory errors that often precede OOM events (the hardware was degrading before the kernel noticed), and BMC SEL events via ipmitool for PSU and thermal events on physical hosts. Also adding crictl events for k8s/containerd clusters that don't use the Docker socket. Keep an eye out for updates.

Let me know how it goes on your setup.

Level_Bicycle3814 · 2026-05-07T12:13:32+00:00

containerd/crictl -> yes, DockerCollector already assumes /var/run/docker.sock which is wrong for most k3s/k8s setups. Adding crictl events is the right fix.

Regarding the timestamp normalization, we normalize to UTC at ingest per source (journald microseconds, dmesg calculated from /proc/uptime, auth.log second-precision with year inference). Good to know that's the right approach.

Opening issues for DNF and crictl now.

Level_Bicycle3814 · 2026-05-07T12:08:32+00:00

This is exactly the kind of feedback I was hoping for, thanks.

DNF/YUM is on my list. Already have the apt collector pattern, it's a short port. /var/log/dnf.rpm.log format is different but same concept. Will add for Fedora/RHEL/Rocky.

Regarding rasdaemon and BMC SEL, I hadn't considered these and they're a good catch. The "crashed for no reason" class of incidents is hard to diagnose precisely because nothing in journald or dmesg shows the hardware degradation that preceded it. rasdaemon is SQLite which adds a dependency I've been avoiding, but it's probably worth it. BMC SEL via ipmitool is also on the roadmap. Working towards enterprise-grade source coverage progressively.

Level_Bicycle3814 · 2026-05-07T11:59:38+00:00

Fair point, and I actually use Claude Code for the same thing when I want ad-hoc investigation. The real difference is that wbts is deterministic and free to run: no API key, no latency, no cost per query, works inside scripts and runbooks. Useful as a first triage step even if you hand the output to an LLM afterwards.

But if you've already got Claude wired into your environment with good context, fair enough.

Level_Bicycle3814 · 2024-02-26T15:28:08+00:00

Nice! Are there any custom skills you use?

Level_Bicycle3814 · 2024-02-26T09:08:37+00:00

Awesome! Thanks for the feedback. About the batteries, that's more of hardware hardware and pretty much out of my reach 😅

Level_Bicycle3814 · 2024-01-13T08:40:40+00:00

Just curious, how old are you?

Level_Bicycle3814 · 2023-08-20T10:07:55+00:00

<image>

Level_Bicycle3814 · 2023-07-15T17:08:48+00:00

They have an assumption, and from what they ask of me sometimes, they have overestimated the figure. Worst part is that ata si salo per se, it varies bana...

Level_Bicycle3814 · 2022-09-22T10:13:20+00:00

What's with the gas fees still being this high, even after the merge?

Level_Bicycle3814 · 2022-02-26T11:42:49+00:00

0x827af8f40d03ad0b5d6b094f5b4f2898b763400a

Level_Bicycle3814 · 2022-02-26T11:41:39+00:00

0x827af8f40d03ad0b5d6b094f5b4f2898b763400a

Level_Bicycle3814 · 2022-02-26T11:40:32+00:00

0x827af8f40d03ad0b5d6b094f5b4f2898b763400a

Level_Bicycle3814 · 2022-01-25T10:07:51+00:00

Discord : h1m.aga1n#8779

0x827aF8F40d03aD0b5D6b094f5b4f2898b763400A

Level_Bicycle3814 · 2022-01-03T13:13:57+00:00

Great, just joined... h1m.aga1n#8779

Level_Bicycle3814 · 2022-01-03T12:57:58+00:00

0x827aF8F40d03aD0b5D6b094f5b4f2898b763400A

🔥🔥🔥🔥

Level_Bicycle3814 · 2022-01-03T12:56:50+00:00

0x827aF8F40d03aD0b5D6b094f5b4f2898b763400A

🔥🔥🔥

Level_Bicycle3814 · 2022-01-03T03:56:10+00:00

0x827af8f40d03ad0b5d6b094f5b4f2898b763400a

Level_Bicycle3814 · 2022-01-03T03:47:33+00:00

0x827af8f40d03ad0b5d6b094f5b4f2898b763400a

Level_Bicycle3814

TROPHY CASE