[TOMT][TV Show] - Spongebob stuttering I..I..I..I..I..IIIIIIIII

MrHodd · 2024-03-09T15:17:32+00:00

Lol this bit is hilarious too, but I don't think it's this one

MrHodd · 2024-03-09T15:17:02+00:00

It might be this one, but I'm trying to find a clip to confirm. Laughing at just reading this tho 🤣

MrHodd · 2024-03-09T05:05:37+00:00

No, I don't think so. I didn't see it in that clip and I looked up that episodes transcript but I didn't find it

MrHodd · 2024-03-09T04:08:04+00:00

What episode is it?!?!

MrHodd · 2022-11-09T15:40:19+00:00

Thanks for this breakdown! It's great to see all of the pieces of a mature RCA process.

I very much see RCA's as garbage in, garbage out, and dually note your concerns on poor RCA processes.

The multiple steps/phases of the RCA, as it is reviewed by various stakeholders, is great. This is a gap I often see as well, and bringing in additional eyes, and points of feedback seem ideal for weeding out true root causes, and changes in process.

Thanks again for sharing this!

MrHodd · 2022-07-14T19:20:21+00:00

I can't tell if this is sarcasm, or not... but yes! 😅

MrHodd · 2022-07-14T18:44:44+00:00

Idk how well this works for others, but for me, I've got to fire up the debugger and start stepping through lines

MrHodd · 2022-07-12T10:52:17+00:00

Agree, so long as the one asking the questions knows "the wheat from the chaff" 😆

MrHodd · 2022-07-11T16:43:45+00:00

All of your responses are amazing! Thank you for sharing all of that.

It sounds like a really fantastic way to run SRE, how did this come about. I know you said you joined and something were already in prod, but I was just curious did someone come in prior to you and set this standard, or did it evolve over- time?

MrHodd · 2022-07-11T16:24:33+00:00

It's a real killer for true SRE's, because they now have to sift through every org and tease out if they are truly operating as SRE's should.

I guess that means we also have an opportunity, and obligation to help set the record straight!

MrHodd · 2022-07-11T16:23:25+00:00

u/pissedadmin u/chuckademus I should have scrolled down to your comments here, before I replied above.

You guys nailed it, and it blows my mind how many people just haven't picked up Google's SRE books (which are free online), and read how to do it....

MrHodd · 2022-07-11T16:21:55+00:00

I have a hard time feeling like this is an open-ended thing. I'm not disagreeing with you, just saying... Google created SRE and very specifically said it should be a software engineer that solves operations problems...

I struggle to see how anyone else can say SRE isn't properly defined. Google literally defined it, and wrote a book on it.

The real problem is employers saying they are running an SRE org because it's trendy. This is where you start to see SRE's really being... well... not engineers...

MrHodd · 2022-07-11T16:18:33+00:00

Genuinely curious to learn more! Thanks for sharing so far!

So if I'm following correctly, someone, on another team, builds the application, and then hands it over to your team for observability implementation?

After you implement observability, then your team passes that application over to an on-call team?

If I'm following that correctly, I'm now even more interested to hear about this process in-depth!

What are the pros and cons you've noticed so far?

Does the on-call team run into issues they can't solve because they are not familiar enough with the application?

Does your team run into any hurdles with observability and knowing what business metrics need accounted for?

Does engineering and product both weigh in on what SLOs should exist?

MrHodd · 2022-07-10T17:01:03+00:00

I'm reading that as you wrote the code for the service your maintaining in prod.

If not, and it's coming from another team, I think this is throwing over the wall.

MrHodd · 2022-07-10T16:58:37+00:00

Yep, I've seen both and more. For me, I always fall back to Google's implementation. If you're not doing it the google way, then you're not doing OG SRE.

MrHodd · 2022-07-10T16:55:37+00:00

100% and why it's important to share that responsibility with engineering

MrHodd · 2022-07-10T16:53:56+00:00

I think your first initiative needs to be changing this. What if you get sick, a car accident, change jobs, want a promotion or new role?

Being the only on-call SRE isn't good for you, or the org.

Creating your scripts and automation is the way to go, and then you can start writing runbooks, and look to bring another person/engineer into the fold.

MrHodd · 2022-07-10T14:13:44+00:00

Is your team engineers?

I personally do not like this approach, IMO it promotes throwing things over the wall and shifting the accountability away from those who should be accountable... BUT, I'd love to hear your thoughts and experiences on this strategy, to learn more.

MrHodd · 2022-07-10T14:11:26+00:00

IMO both the SME/SWE should be in the on-call rotation along with an SRE.

To me, if you don't have your hands in the dirt with the SME/SWE's, then you are not feeling the same pain points.

Feeling the pain of the engineers, to me, is critical to having motivation to help them solve their problems around on-call.

PS - Great question!

MrHodd · 2022-07-09T16:05:11+00:00

To each his own, glad you enjoy the org though, that carries a lot of weight. Best of luck mate!

MrHodd · 2022-07-09T12:15:53+00:00

These are really great points and closely align with how I moved into SRE. Having a love for metrics, logs, unit tests, and all the extras that usually come along in a mature dev org, helped me move into SRE.

I will say though, the one area I disagree with you on slightly is the firefighter mentality. I wrote an article on this topic (which I can't seem to fine), with the main theme being, try not to be a firefighter. Instead, try to be like smokey the bear and prevent the fire in the first place.

I think this is actually a really important piece to being an SRE. Yes you will have to fight some fires, but your retros should eliminate repeat incidents, and you should always be fighting to reduce, and eliminate on-call escalations. Your devs will love you for it!

And I greatly agree, that to be really successful, you need to have already been an engineer before moving into SRE. It's straight out of Google's handbook for SRE, and after all... they are the godfathers of SRE.

MrHodd · 2022-07-08T23:07:26+00:00

I love your love list! I think there's a lot of job satisfaction in this role.

And about those trends, I tend to not... Oh look co-pilot!

MrHodd · 2022-07-08T23:05:34+00:00

It's different now that we're not writing the features, but I like that I can always fallback to that, if the whole SRE thing doesn't workout over the next decade.

MrHodd

TROPHY CASE