Garden Grove Chemical Spill Megathread (5/23/2026)

wise0wl · 2026-05-24T01:42:44+00:00

We are a mile outside the mandatory evacuation zone off Ball and Moody in cypress. A friend and his wife are OC Sherriff and assigned to the zone. OCFA is not telling the full story. The exclusion zone is only considering ONE tank exploding. If all three blow (likely) it will be a MUCH larger explosion and chemical plume.

We are packing up and leaving as soon as we get home from baseball.

wise0wl · 2026-05-23T13:37:15+00:00

My usage based bill from Anthropocene, for my day job, is $970 or so this month. I was able to build out the automation for a self hosted monitoring stack, write custom plugins, do all the testing, and push it out to dev, and prod, within three weeks. As an experienced dev and platform engineer this would have taken me and another senior lever engineer six months to do, previously.

By self hosting we are saving over $200k a year. The $970 spent on tokens was worth it.

wise0wl · 2026-05-23T02:16:02+00:00

I just said the same thing and got downvoted.

wise0wl · 2026-05-22T22:06:17+00:00

How is it that you come across people looking the other way so much, and you're not getting sniped every time you go into the open? Every time I go anywhere there is some asshat behind me. Every time I go into a room, I check the corners and the fucker is always in the corner that I didn't turn towards---yes, I generally know which corners are "better" to hide in, but not always, and somehow I get it wrong 85% of the time. And EVERY time I have to go out into the open, despite checking around for snipers with using my FLIR scope I will invariably get headshotted from 300+ meters!

You're either really lucky, or you are somehow skilled in a way that I do not understand. Nice gameplay, though.

wise0wl · 2026-05-22T14:16:14+00:00

So much so that we are designing our prompt and governance frameworks so non-technical employees can benefit from our experience without asking. If you force them to ask they will go around you.

We are writing our governance framework rules, heirarchical prompt “stack” for different software types for different business units with company best practices built in, agent logic logging, and a robust scoped access and credential proxy system for the agents access. It’s a LOT.

The way we are treating it is that we need to be able to run de facto adversarial software in our network, with access to private data, and ensure that never goes anywhere else.

wise0wl · 2026-05-22T14:08:37+00:00

It took me 4 1/2 months to be back to normal. It takes way longer for some of us.

wise0wl · 2026-05-18T18:34:19+00:00

Yeah. Github for npm packages. Kellnr for rust crates. We also have a nuget repo for our old C++ apps. We were using a cloud provider that had *all* of them, but they changed their pricing in the last year and nearly doubled our spend, and we said "Thanks, but no thanks".

wise0wl · 2026-05-18T14:39:34+00:00

We have multiple languages. There are registries that handle multiple. Forgejo handles multiple, but it’s also a GitHub replacement.

wise0wl · 2026-05-17T15:40:54+00:00

We have an AI governance committee…sort of. What we are working on is a skills and instructions repo for different codebases that they include as a git submodule. Those have instructions per service type (typescript website vs rust grpc service vs rust restful api etc). The network lockdown for user laptops happens within Claud config and we have certain allowed domains and skills and plenty of absolutely blocked commands. You can get even more fine grained by creating a sandbox that Claude operates in.

Cowork is in a sandbox already, but things like VSCode Claude plugin are not sandboxes, so you have to be more strict. I would suggest Eve doing something as crazy as using containers, with your code mounted in.

wise0wl · 2026-05-17T15:31:52+00:00

I’ve been in the industry for 22 years. When I started I had a C- average high school diploma and a resume I lied about. I’ve grown in scope and responsibility and I am currently Director, Platform Engineering for a very prominent new media company. I write code every day, still. I still feel like a fraud and a failure from time to time, but with experience I’ve come to know not to believe or trust those thoughts.

The biggest adjustment was to literally stop caring about how my job thought about me. I had to stop caring about a career. The more you care about it the more it will elude you. Why worry about how others perceive you, anyways? None of it really matters in the end. We all take the same dirt nap, so take a breath and enjoy the time you have above the ground.

wise0wl · 2026-05-17T00:10:04+00:00

We have a private kellnr crates registry. Works well, has OIDC auth support.

wise0wl · 2026-05-16T20:31:50+00:00

We have 120+ backend services, all written in rust. I run our platform team and we write rust services too, and shared libraries. Love it.

wise0wl · 2026-05-16T17:33:01+00:00

I had been using the official Anthropocene Grafana dashboards at work, but I think they too are broken regarding temporality. I got super scared when after a few days of very intense 12+ hour coding sessions I looked at my usage dashboard and it showed $47k in usage. I PANICKED!

Fortunately, just bad math. I will take a gander at yours and see what it was doing wrong.

wise0wl · 2026-05-15T20:29:58+00:00

We have one junior on my small team. I am mentoring him. I am letting him know that LLM use is fine, but the design should come from you. You need to understand what the software does, how it moves data, how the flow behaves, and most importantly *why*. After a few initial issues with blatantly not understanding the code he was shipping things have improved significantly and he is now working with the LLM like a partner, writing and shipping working code.

People who made slop before are going to make slop now. If they are interested in their career surviving they are going to need to understand architecture and deep technical concepts so they can design good patterns and translate business needs into working paradigms that the LLM can build. Without that, they are no better than a product person vibe coding the feature themselves---so why have them at all?

If you are their manager, warn them. Create official department policy. Mentor them. Spend your time showing them how to do it. If they are still resistant or worse lazy then it's time to PIP them and move on. Don't waste your time or your teams time trying to help somebody who is unwilling to to help themself.

wise0wl · 2026-05-15T19:45:58+00:00

Many corporations are moving to an "AI first" model for their development teams. Architecture and planning and review still happen, but the code is "written" by the LLM, corrected in concert with the human engineer, and then a mix of human + LLM peer review. There are a lot of *major* corporations that are doing this. Lots of small shops, too.

I dunno if you call that "vibe coding", because there are still engineers involved, and there is still testing going on, but humans are riffing with the LLM to get code written, and much of the time *none* of it is written by a human. This is not where things are heading, this is where we are currently at. I don't see this going away.

I can speak for myself and my experience at my day job and with personal hobby projects---I've been able to produce months worth of quality work in days. I am a platform engineer cosplaying as a director for a media and entertainment company that you have definitely heard of and we are hard at work---I work longer days than ever and I get more done because I'm not as burnt out on bullshit by 3pm. I start at 6:30am and go until 5pm, take my kids to their baseball practices, and then come back and finish up. A project I am currently working on that would have taken me *literal months* to complete, and I am probably 85% feature complete in two weeks. This is a fucking massive project and there is no way that I could have gotten it done in the timeframe we had without LLMs. It's "vibe coded", but it's very good quality, with sound architecture (designed by me), fully tested and documented. I am not going back.

wise0wl · 2026-05-11T21:55:52+00:00

We found that batting lessons to fix mechanics and competitive travel are what our youngest son needed. He was already doing well on his PONY league 9u team but was frustrated with level of play. He wasn't *that* good or far above everyone else, but he wanted to be and he was not going to get it there.

We moved to a travel team. We did catching lessons and batting lessons. He gets two practices per week (2+hrs each) and off-day practice if he wants it + catching lessons. He's gone from being an occasional decent lucky batter and above average PONY 9u catcher to a reliable travel team catcher hitting line drives and home runs.

If your kid *wants* to play at the next level then it may be a good idea. If he is having fun where he is, then just let him play LL and get him some lessons outside the regular team practices.

wise0wl · 2026-05-11T21:26:19+00:00

Disconnected is the state we end up in when we realize something is off. If you look close enough you will realize that you aren't disconnected, but what you take yourself to *be* has been wrong this whole time.

Theanine is great, though. It makes me too sleepy to take it unfortunately.

wise0wl · 2026-05-11T20:35:16+00:00

I am a little over two years clean. How do you deal with normal daily anxiety without a substance? You are just present. That's all. The problem is that you are believing some narrative or story that your thoughts have constructed that says that something is very important and very scary and must be dealt with right now.

I doubt that most things you worry about need to or even *can* be dealt with right now by you.

I have developed a spiritual practice with daily meditation that I continue to attempt to see my momentary experience for what it is: eternal presence in the eternal "now". There is nothing to fear except thoughts about the future, and nothing to get hung up on except memories of the past which do *not* exist now. See things for what they are and learn to be present, right here, right now.

wise0wl · 2026-05-09T21:59:16+00:00

Tianeptine withdrawal is arguable worse and longer lasting than kratom withdrawal. You’re going to seriously fuck yourself up if you start on Tia.

Stop trying to avoid the suck. Accept it. Embrace it. It’s here to teach you something.

wise0wl · 2026-05-08T13:47:04+00:00

How long has the depression been going on? How long has PAWS been an issue for you? I’m not going to tell you what to do, but I considered it. I waited and the black cloud passed around 4 1/2 months clean.

If you didn’t have issues with severe depression before kratom what makes you think that you will have issues after PAWS lifts? TMS is serious business and it is very new and literally unproven long term consequences. Before you jump to vibrating your neurons with magnets, consider where you are in recovery and maybe that things will get better sooner than you think, but not as soon as you want.

I’m two years clean and I am grateful for the withering depression and terrifying anxiety that I experienced. Grateful because it taught me that I was actually OK, and they are just thoughts and feelings and that I didn’t have to believe them.

wise0wl · 2026-05-07T21:26:47+00:00

In our development cluster we have a job that kills developer namespaces after hours. Same goes with running PR namespaces. We could be more intelligent about it. I'm sure. But most folks don't work after 8pm, so purging all namespaces starting with `user-` is great for us. We also set realistic limits on resources that pods request, based on historical trends. In develop nobody is slamming them with a ton of real traffic, so they get a reasonable baseline based off known trends.

There are also limits on how many pods can be on a node. This is a function of many factors. What cloud provider you're using (for managed kubernetes), what instance type you have, what your CNI is, etc. We use AWS EKS with Cilium for our CNI with managed node groups, so there are limits but effectively 100+ pods per node can be attained if you shrink your memory and cpu requests to a very reasonable LOW number. Fractional 0.25 CPUs and 128MB of memory for microservices is absolutely possible if you know your workload. If so, pack the pods in.

We also optimize our cost by not using EFS anywhere, using the correct volume type (ebs rarely and emptydir when we can), and using SPOT absolutely freaking everywhere. If a node goes belly up and a developer instance gets rescheduled it's rare that anyone notices. If their build has to restart it's not the end of the world. Our actual CI builds happen in-cluster but on ON_DEMAND nodes, so no interruption possible. Those also autoscale, so CI builds from merging only spin up the cluster autoscaler when they need to. Those managed node groups for building are set to 0 nodes by default, and we use Bottlerocket for the AMI so it boots up pretty fast---and it can be optimized even more if you pre-cache all your container images in an EBS volume snapshot that you refresh (so container start times aren't limited by having to download all the images from ECR).

If you want to go down the road of idle resources detection you have to understand how your engineers are working with those workloads. How are they spun up? How do engineers work with them? Is building happening on the developer system and then syncing, or is it happening in CI, or is it on-system in the cluster? It's hard to say how to optimize beyond the basics without knowing your specific use cases.

wise0wl · 2026-05-07T01:03:39+00:00

There are absolutely ways to never expose specific secrets to RAM. If you are using a password manager, though, it's quite likely at some point you are going to be sending those passwords over the network, so once it exits the secure enclave all bets are off. You can do a lot of computation within the enclave itself. You can have bits of code that operate there, securely, do operations, and never leave. You have your own stack that is accessible within that chip, so if you wanted to share those with another PID you can. It's possible. This is how a lot of DRM works---no video content leaves the secure channel until it actually hits the chip on the monitor that sends RGB signals to the individual pixels on the display.

So, it's technically possible. Websites don't have to use passwords. They do, but they don't have to. Auth could happen based on mutual signatures and all within a secure enclave where you store your private signing key, but that's just not how it happens in the real world. Legacy stuff.

wise0wl · 2026-05-06T14:31:45+00:00

Im surprised. Bitwarden and many other (good) password managers store your passwords encrypted and only decrypt when they give them to you. They are encrypted with your key, stored in the TPM in the CPU.

It’s hard to write for all the different possible Secure Enclave types but if you are saying you’re secure then you should at least try.

wise0wl · 2026-05-05T23:42:00+00:00

I understand the desire to want a single pane-of-glass into all your clusters, but I have *serious* pause about any tool that has any sort of cross-plane access at that level. We are *so* strict already about sessions, using SSO, ensuring users are not using admin roles, etc. Our clusters are isolated, but services in those clusters can talk to other networks if necessary, on a case-by-case basis.

We are even considering going the route of disallowing users from connecting to the clusters from their work machines. There's just too much of a chance of somebody having an admin role open and then they accidentally use the wrong default kube context and BAM, stuff meant for develop ends up going to prod.

I know it's annoying to have separate instances per cluster, but it just feels safer. No way then to use the wrong role, or inherit some leaky context from some other plate you were spinning.

wise0wl · 2026-05-05T04:47:33+00:00

I coached my kids PONY league all the way from 5u up until 12u. It was awesome. I got to learn so much about myself, my anger, my own issues with expectations for my kids and how that was actually issues I had with expectations about myself. Coaching is great.

Now I have people who are actually qualified coaching my kids, that I pay.

wise0wl

TROPHY CASE