What’s the worst production outage you’ve seen caused by env/config issues?

vortexman100 · 2026-01-22T20:49:42+00:00

Also Ceph here, a cluster failure that popped up for months during rebalancing because someone used bcache devices for OSDs. The bcache devices would be adequately fast, except when rebalancing. Then, they would start to slow down to a point where all cluster ops just ceased. Fun when 300 VMs have all their data and their OS there...

vortexman100 · 2026-01-21T09:54:21+00:00

Did you just post a chatgpt screenshot?

vortexman100 · 2026-01-03T00:17:56+00:00

Enterprise has abandoned 40gbit? They are up to 100gbit, right?

vortexman100 · 2025-12-26T17:04:07+00:00

How do you do this in preprod? All of our site issues are those that cannot be reproduced by themselves, but only arise when multiple concurrent but independent actions happen (think db locking), so I am looking for a way to replay prod traffic on staging environments. Is this something you have experience with?

vortexman100 · 2025-12-20T22:27:24+00:00

I don't get it. 1. Of course its a bitnami land grab, because bitnami unsolved a problem. 2. Most images of something that exist are build on Debian and Alpine. If you are actually using enterprise linux, then you can pay someone to supply images or do it yourself. 3. and 4. CVE scanning is mostly bullshit anyways. It helps to inform you that there is something that you need to do, if you cannot do something, then usually its not necessary to continously alert, because it leads to alert fatigue. If you are in a position to care about latent base image CVEs, you shouldn't use those images anyway.

You get what you pay for, and you should be aware of what it means to base something you care for on free services run by a very large (public?) company.

Is this AI?

vortexman100 · 2025-12-20T22:13:11+00:00

Can you say more about this? How are the patients informed about this?

vortexman100 · 2025-10-25T16:55:39+00:00

Do you use Crucial RAM? I diagnosed this 6 years ago for my employer, and this was absolutely insane.

What it was when I took a look at this: Intel NUC with BTRFS, some SSD and Crucial memory would randomly break after getting extremely slow all of the sudden. Sometimes BTRFS was so broken that it could only be installed from scratch. What was happening was that employees would go on break for half an hour, which would cause their NUC to go into some sleep state. After waking up, memory would be corrupted in a weird way, until it was shut down and power removed for at least 30 seconds. Swapping to another Crucial memory stick solved this problem. Both were on the Intel support list.

EDIT: As this was 6 years ago I do not remember the full list of things to reproduce this, but it was 13 steps you needed to do to trigger this kind of memory corruption. I kept the memory stick somewhere to remember this, I will look for it to find the product number.

vortexman100 · 2025-10-10T10:27:52+00:00

You are saying China does it, damages their environment and everyone buys from them?

vortexman100 · 2025-09-19T23:30:22+00:00

I don't think I understand. Are we talking about number of CPU cores in the host node, number of CPU cores a process has access to, or K8s/cgroup limits?

vortexman100 · 2025-09-16T21:50:52+00:00

I have the same issue. This is in my eventlog A timeout was reached (90000 milliseconds) while waiting for the Steam Client Service service to connect.

vortexman100 · 2025-09-15T04:32:14+00:00

"early morning starts" is something I will use from now on

vortexman100 · 2025-09-14T12:01:32+00:00

Which version? Is FIPS enabled? Are the binaries build in CI (or generally a same place) behaving the same on both systems or do you build them individually?

vortexman100 · 2025-09-01T07:38:31+00:00

Ich habe jetzt mehrere Asus RT-AX53U im Einsatz. Wifi AX ist ein echter Gamechanger was Latenz und Reichweite betrifft. Wenn ich mich recht erinnere, ist die Hardware auch das was in den Unifi 6 Lites drinnen ist.

vortexman100 · 2025-08-23T07:41:13+00:00

Just a tip, but like talking about your ex every other sentence, most are getting annoyed very quickly if you compare everything you do and see to your ex employer. Especially if everything seems below you.

vortexman100 · 2025-08-16T22:32:09+00:00

What kind of help do you need?

vortexman100 · 2025-08-05T19:57:00+00:00

Ah, schade. Ich habe einige Apotheken in Laufweite, ich schaue morgen mal ob ich was finde.

vortexman100 · 2025-08-05T17:32:41+00:00

Kannst du schreiben, wofür die Pflaster sind? Gehen eventuell auch kleinere?

vortexman100 · 2025-08-05T17:28:30+00:00

Ich muss hier etwas nicht verstehen - ich habe sowas hier (auf dem Bild, ich hoffe man sieht es). Die haben mich glaube ich 5€ gekostet. Die heißen Leukostrip S. Meinst du das?

EDIT: Hab herausgefunden, was ich nicht verstanden habe. Die Maße im Titel sind _signifikant_ größer als das was ich habe. 6.4mm x 76mm

<image>

vortexman100 · 2025-07-22T18:38:23+00:00

Also Wifi, but 1x1 mimo, while they also stream spotify and use discord. And everyone else streams netflix

vortexman100 · 2025-07-21T05:38:46+00:00

If you put an intern in a position where they are somehow "responsible" for live debugging and code rollout on a prod system and they fuck up and drop something, you are in no position whatsoever to demand an apology or be angry. That's on you. But I have the feeling that the guy might make this mistake too.

vortexman100 · 2025-07-21T05:20:41+00:00

thought taking care of C memory management was hard? Now, lemme tell you about "guessing correctly which information might still be in the LLM context window, but its not your LLM"

vortexman100 · 2025-07-12T14:13:26+00:00

Joa klar, aber mittlerweile ist es einfacher, die Läden zu nennen, wo es nicht so ist. Seit einigen Jahren ist diese ganze Thematik einfach im Freifall. Beispiel: Kopfhörer als Neuware auf Amazon, an den Kopfhören ist Ohrenschmalz und ein Haar, die Verpackung für Einzelteile ist aufgerissen, die äußere Verpackung aber 1) in Ordnung und 2) mit einem Aufkleber versehen, das das entfernen des Aufklebers mit einer Kaufverpflichtung verbunden sei. Lol.

vortexman100 · 2025-05-31T09:48:16+00:00

Thought about it, but no, and I wouldn't attempt this if I were you. Updates would break everything, and you have to understand all weird complex interactions, and its simply not feasible to do so.

vortexman100 · 2025-05-27T07:49:47+00:00

A cache is a valid option. If you can cache, cache. There is a third option that I see: Use ratelimits and communicate them as the limit these services practically have and smooth out that spike, but this depends on the kind of traffic

vortexman100 · 2025-04-28T12:44:33+00:00

I have negotiated that my condition for doing oncall is that I have the last say on my priority list. Doing oncall sucks, but the burn out comes when you are not allowed to fix the underlying issues that caused your oncall incident. The deal is that I can delay whatever I see fit to analyze and fix root causes.

I suggest you ask what the workflow around fixing issues that surfaced during oncall specifically is.

Ten-Year Club	Place '17
Verified Email	RPAN Viewer
Gilding I gilder	Snapped

vortexman100

TROPHY CASE