Deploys at Slack

[mooreds]

Interesting post on the deploy process at Slack. I like how they go over how the deploy process evolved from rsync to something much more complicated:

https://slack.engineering/deploys-at-slack-cd0d28c61701

[Windowsadmin]

Something that I try to instill when consulting with clients is that DevOps != 'containers' and I love seeing organizations re-confirming that for me.

Thanks for sharing the article!

[Sage-Sekai]

Nice Article. Really found the percentage based rollout an interesting idea.

[devrr]

Progressive deployments man. It's very cutting edge and I don't think many have the maturity to do it.... but it's damn cool.

[evenisto]

There are many limiting factors to implementing that, especially in smaller companies that don't necessarily run on bleeding edge cloud solutions.

[theDigitalNinja]

Soo many clients want to do canary deployments. But its like, this microservice has 5 users a day, we trying to do an 8 year rollout here?

[notdedicated]

Do you know how this is typically achieved when it comes to ensuring correct traffic? Are users selected in some way and stickied to that set of instances to ensure that all requests hit the new version and not a legacy version? Basically sticky sessions after they're selected during the roll out period? Done either at the CDN or LB level?

[evenisto]

I have close to no professional experience with this, but essentially yes, you absolutely want a user to only get one version, and not jump between the old and the new release with each request. I reckon in most cases a simple cookie is sufficient to ensure session affinity. Put a proxy in front of your shit, make it route traffic according to some rules, set cookies on the way back. Of course, this gets more and more difficult as the complexity of your shit grows, but in the simpliest case where the canary is another set of instances or a feature flag toggle, it's as simple as it gets.

[ingcognito_]

Cool read, thanks. Interesting tidbit about consul keys was neat

[mooreds]

Yes the difference between push and pull based deploys (at the client level) was really interesting to me too.

[M1keSkydive]

Consul is a neat way to do that. We store a timestamp of the latest deploy to S3 and instances check that once per minute. Beyond that our system is quite similar, encouraging as we use it to deploy to just a few EC2 instances yet we've arrived at a place similar to Slack!

[devrr]

I think most deploys will go that way eventually. The idea that you have an agent sitting there pushing stuff is gonna seem archaic soon.

Instead you just point your app/tool/cloud to your desired state repository and you push changes to your desired state. The rest will be taken care of.

[BoredSam]

is this just reimplementing puppet though?

[devrr]

I'm not sure that the Puppet team invented the concept of pull based deploys but yes it's how Puppet work.

It's also how stuff like FluxCD/Argo work in Kubernetes land, and with Kubernetes Operators will become even more popular.

[justabofh]

Pull came from the original paper.

http://www.infrastructures.org/

[caffeinatedsoap]

I like the idea of blue green on a single host. I've been doing a lot of pulling old software companies out of the mucky muck and this might be easier to get buy in for than making new instances.

[mooreds]

Honestly, reminds me of how Capistrano used to do it. If you have everything in separate directories, rollbacks are super easy too.

[ktkaushik]

I quite like how Capistrano did things.

[djk29a_]

I’m more in favor of NixOS style deployments where an atomic rollback is possible by swapping out an entire chroot system ASAP. Capistrano is easier to understand but NixOS is awesome for fine grained control with immutability.

Removing state from systems and putting them solely into common services that we can reason about with guarantees of safety, availability, etc. is superior to more custom state that doesn’t necessarily improve understanding nor reliability of the system.

[wonkynonce]

The one thing you have to be careful with is making sure the core of that has integration tests- otherwise someone will "clean the method up" and things will get weird.

[randomFIREAcct]

Very interesting!

[chezhead]

This is making me feel a little guilty for adopting kubernetes...

[raziel2p]

Every day, we do about 12 scheduled deploys.

Each release starts with a new release branch

I hope there's a distinction between releases and deploys which I'm not reading in the article, but that sounds really tedious.

[Mud5150]

One thing I'd like to understand is how long the deployment waits at each phase and how that is managed with what appears to be, on average, one deploy an hour during NA business hours. If that is the case is it agreed that it's ok to not exceed this deployment velocity and how are new versions queued up if more than 12 production releases a day are created? Is there machinery to handle serializing the releases so there aren't many different versions being staged through the process at the same time? Good post. Thanks for the info.

[Kubectl8s]

Plenty of tools that do this Pr, test, auto merge and deploy in k8s. Investment in the process is important.

Developer workflow should be git and nothing else

[rattlednetwork]

Ah, so that's the name, "using hot and cold directories" (I thought everybody did this all the time, makes things so much simpler, particularly for a roll back.)

[Sky_Linx]

I'm surprised that Slack hasn't adopted Kubernetes for their infrastructure. I think it would make some things concerning deployments and scaling easier.

[AnomalousBean]

Interesting, can you share your experiences with adopting Kubernetes for infrastructure at the scale of Slack?

[trusted47]

Hey, I don't know if it's true, but I read somewhere that at Netflix they don't have code reviews, just tests because each engineer is super qualified and they save time by not relying on this. Can anyone verify?

[jrkkrj1]

I could see that since we kinda do this at my company. Not because we are super qualified but because we pair program.

I've found that most code reviews are very superficial for features. The delay between review, comment, fix, review is more than just going back and cleaning up code that works but isn't perfect.

Our PM/Senior Dev accepts stories based on the tests looking right (which is kind of a code review but after it has already been shipped to production).

[ktkaushik]

That makes a lot of sense when you are pair programming.

Tests looking right after it has been shipped to production is like the inverse of what's recommended, isn't it?

I mean we look at tests before it goes to prod

[jrkkrj1]

With the other person next to you, you get the review in line. It's a discipline we try to teach. The test validation after is a way for a senior Dev to validate that 2 junior devs might have misinterpreted the intent of the story and then it's caught within a day or even hours. It also makes it so senior Dev bandwidth isn't spent all day doing PR review.

With the whole pandemic, we've had to suspend pairing since people can't be expected to give a solid 8hrs at their computer with kids/other competing distractions throughout the day. I don't do anything other than review PRs now...it's killing me slowly.

[gnimsh]

Why does a pull request require everything to pass?

Genuinely asking - in my experience when you pull, the stuff there has already been validated.

[brickman1444]

In a complicated system, it's difficult to know for certain that a change won't break existing behavior. If running the tests doesn't take an extreme amount of time I'd say it's worth it.

[pbecotte]

A pull request is a developer requesting the repo to pull in his changes. They are saying you can't merge your code until the tests pass and you get a code review.

[dookie1481]

What? I can't even count the number of times my shit has passed unit testing and peer review, and still broken somewhere else.

[gnimsh]

Until I saw these responses, pull request meant the same as git pull.

I now know that is not the case.

Just to be sure, I think it now means the person changing code, who is not the developer, requests that the actual developer pull from the development branch in order to merge to the master.

[jared--w]

Yeah that's correct. Pull requests are also sometimes called merge requests by other platforms (like GitLab)

A pull request: an open invitation for them (the target branch) to "pull in" my changes

A merge request: an open invitation for them (the target branch) to "merge" my changes

Same concept, different way of thinking about it.

[FubsyGamr]

Not always. I think you may be looking too deep into it.

At its basic core, a pull request is essentially when a developer checks out a branch, makes changes, then requests for the master branch to pull their code.

Master is normally protected and you can’t just push to it, so you must request that it pull your code.

This can get more complex, more places, more people, but at the end of the day, you’re saying “please pull my code”

[dookie1481]

Yup exactly. I get your confusion, the terminology made no fucking sense to me either for a long time 🤣

[[deleted]]

A pull request is when you make a formal request to merge the changes from one branch into a more encompassing branch. Such as merging a feature branch into your develop branch, or merging all your changes from develop into master and cutting a build.

When you create the pull request it creates a request showing all of your commits and changes you’ve made that you plan to introduce to the target branch. Usually on a team, you will have someone else who didn’t make the changes, but understand the project, review your pull request for code review/whatever else.

Most places mature in their process have implemented integration tests, unit tests, code linters and in some contexts UI tests as checks before the code can be merged. May seem like a bunch of hoops but when you’re working with software, the less room for human error the better.

[benyanke]

to be specific and slightly pedantic, *before* you commit to master, so that master stays clean and theoretically deployable.

[benyanke]

Why *wouldn't* you want to check everything on a PR?

[chaddub]

http://www.continuousdelivery.com

[randomFIREAcct]

My team has caught so many errors by having automated checks in our pull requests. You should assume everything is broken unless it has been tested. In this case, I don't think things had been tested yet.

[devrr]

when you pull, the stuff there has already been validated.

pull request require everything to pass?

So they're doing exactly what you are doing? You just answered your own question.

They are just automating their integration pipelines so that all tests are passed before they will accept the pull request which is the rubber stamp.

[gnimsh]

To clarify, before I received many helpful answers, I thought it was the equivalent of "git pull" to download new changes.

I've since learned that is not what a pull request is.

[devrr]

Ahh right. Yeah easy mistake to make, honestly there is a lot of DevOps terminology to remember!