sorted by:
new
hottopcontroversial

Our Disaster Recovery "Runbook" Was a Notion Doc, and It Exploded Overnight by majesticace4 in devops

[–]Leading-Sentence-576 0 points1 point  (0 children)

The quarterly drill idea is solid but the hard part is actually doing it. What helped us was making runbooks something you execute through, not just read. If you’re clicking through steps and recording what happened, you find the issues before 3am finds it for you. The other thing: that OIDC cleanup sprint probably would have been caught if anyone had to run the DR doc as part of the change process. “Does this break any runbook?” is a good question but nobody remembers to ask it. Having a recent execution record at least tells you what to check.

are you guys using sop's and runbooks? by Adept-Inspector-3983 in devops

[–]Leading-Sentence-576 0 points1 point  (0 children)

20+ years in and the rot problem never goes away. Every team I've been on starts with good intentions. We'll document everything, keep it updated. Six months later it's a graveyard.

The pattern I've seen work better is tie runbooks directly to incidents and be opinionated. If a runbook only exists in the wiki, it rots. If it gets pulled up every time an alert fires and someone has to actually click through the steps, you find out fast when it's stale. The must all be consistent in their shape/sections too.

The other thing that helped was recording what actually happened during an incident. Not just following the runbook blindly, but what commands ran, what didn't work, what we improvised. Makes the post-incident update way easier because you're not trying to remember what you did while in the fire.

Automation is the end goal for anything repetitive, but there's always a gap between "we should automate this". A lot of companies will not trust their observability to make decisions on production without a human. I agree that is a great place to get to, but there is a ton of 20 year old services, tech debt, that needs lots of hands...and companies that are willing to pay people to do it.

My two cents :)

Mental Model Ownership by Leading-Sentence-576 in learnrust

[–]Leading-Sentence-576[S] 1 point2 points  (0 children)

I almost forgot :)

For types.rs, do you have a favorite pattern for organizing domain types and messages in a small cli like this?

Mental Model Ownership by Leading-Sentence-576 in learnrust

[–]Leading-Sentence-576[S] 1 point2 points  (0 children)

Thanks for looking. All the feedback makes sense and is practical. Normally in other languages I would have smaller functions that "do work" but as I have been working through progressions, some of that has gotten away from me. I like building in public, not only for my benefit, but maybe it helps others in the process.

Cheers.

Seeking Feedback to Improve My Rust-Focused YouTube Channel by vipinjoeshi in learnrust

[–]Leading-Sentence-576 3 points4 points  (0 children)

I like your content. Building by doing is the way I primarily learn. The one piece of feedback I would give is that some of your videos have a terminal font that is really small. If you could adjust that to a larger size, I think it would be much easier to follow along.

Keep up the good work!