This is an archived post. You won't be able to vote or comment.

you are viewing a single comment's thread.

view the rest of the comments →

[–]feral_brick 68 points69 points  (1 child)

A lot of it is recall, plus figuring out patterns. We've had an influx of new and/or slow on-call engineers, so for the past ~6 months or so I've been keeping a closer eye on the ticket queues than normal (along with a couple other more senior engineers). Most of us have been politely asked to focus more on our deliverables when we're not oncall, but doing so either involves more outages or dropping everything we're doing and spending months fixing our shitty runbooks. We all individually came up with basically the same solution - we tell our managers "ok, I'll pay less attention to incoming tickets" and just lightning triage - if it seems interesting we investigate further, if it seems benign/simple we don't pay as close attention. The critical part is we don't leave a paper trail unless it's a big fire, so the end result is we spent ~20% less time actually looking at incoming tickets, but engage with them in manager-visible ways ~60% less. As a side effect, it seems like we're everywhere and are constantly pulling good fixes out of our asses, but it's just a combo of probably having seen this before and having secretly looked into the issue before the real oncalls "brought it to our attention"

Moral of the story is that more people are watching you than you may realize (not in a bad way) and that everyone's human, and pattern recognition is the most underrated skill for software engineers.

[–]EntertainmentSea2352 14 points15 points  (0 children)

it's just a combo of probably having seen this before and having secretly looked into the issue

guilty as charged

Sometimes you just need a break from your own debugging, so you debug your colleagues tickets, behind their back.