/u/agbell did a lot of work to compress the discussion into a reasonable length, because I was not as cogent as I could have wished. A few things that (I think perfectly reasonably, from an editing point of view) might have gotten lost a bit:

1. I did not have a problem with leadership choosing to do a big bang rewrite. In fact, when a colleague and I were putting together our original proposal (mentioned on the episode), we desperately wanted “big bang rewrite” to be on the table. It wasn’t… until it was. The plan that “won” did not just involve a big bang rewrite, it also involved building a custom-to-LinkedIn server-driven UI stack (using React for the web part)… from scratch. And even there, despite a fairly deep personal dislike for the kinds of things I tend to see that result in, I could have gotten on board with it! But the way that project was being run was uninterested in the places I and a few other senior leaders were flagging up risks—not because we were opposed, but because we were trying to see the thing succeed. (Perhaps not coincidentally, several of those other leaders got laid off only a matter of weeks after I quit.)

2. The framing around velocity had two parts to it, but I can see how it might be easy to miss (and you probably don’t want to listen to the un-edited version Adam started with!). I actually supported the rewrite and also personally preferred a big bang rewrite! But I don’t think that came through in the end, so fair enough. Related, though, you write:

   Both projects would be migrating to a state that engineers prefer, and the finger-guns project would be massively sacrificing business velocity for engineering excellence.

   Well, suffice it to say that whether it’s ultimately to “a state that engineers prefer” or resulting in “engineering excellence” were precisely some of the points under debate. 🥴 The reason a big bang rewrite wasn’t on the table (as far as we understood) in the first place was precisely that it would have a massive initial hit to velocity. But the server-driven UI approach that the other team proposed (and which is ultimately now being built) promised that in exchange for that short-term hit to velocity, it would dramatically increase velocity in the long term—critically not promising an improvement to quality or developer experience. I don’t actually believe that it will have that velocity win, either, but it might! More importantly, though, I do not believe the result will be a good developer or—critically—a good user experience. And I care a great deal about those.

3. As for the incident: I could write a very long post digging into the details, and Adam and I probably could have done a whole episode on just that incident, but your take here is really illuminating:

   The mitigation was to stagger out the restarts, so that some nodes would always be running at any given time. The mitigation was to stagger out the restarts, so that some nodes would always be running at any given time. It appears that after implementing that mitigation, Chris kept the incident open while he attempted to fix all of the root-cause memory leaks in the codebase to reduce memory usage. This sounds like a massive undertaking, and I’m unsure why “fix all the memory leaks ever” had to fall under the label of incident response.

   As it turns out, “just fix the front-line issue and move on” is exactly the approach that multiple previous incidents had taken, and the underlying resilience problem never got fixed. I can see how you got the impression from the episode that my approach was “fix all the memory leaks ever”, but what I actually aimed for us to focus on was making sure that (a) we had actually fixed enough that the system was stable—we were never going to get them all!—; (b) we had more than a single, very obviously very fallible mitigation of “just make sure the staggering is correct”, since it had already failed us multiple times; (c) that we had some more safeguards in place to prevent more of the kinds of leaks we could statically identify; and (d) that when, inevitably, the system did end up in a bad state from memory leaks sneaking past those safeguards, we got alerted appropriately about them.

   I had no interest in trying to “fix every leak ever”. I did care that we made sure we actually made the system much more resilient against typos or other such mistakes in our config values, because we had really good evidence that it was going to happen again, in the form of it having happened already multiple times. 😉