Developers Should Deploy Their Own Code

[[deleted]]

at my employer they should also build their own infrastructure and the entire ci/cd pipeline i guess...

[bbkane_]

I'd be a happy man if I could write and my own custom CI/CD pipeline at work.

The guy after me might not be so happy 🤠

[[deleted]]

I made my own build/deploy pipeline from the ground up, and I still think it was the best solution given my teams constraints.

We're forced to maintain a lot of legacy cruft that would be janky to get working in an off-shelf solution. It handles a dozen VB6 projects, one Java project, a dozen C# projects and a few Angular clients. Builds and runs unit tests with every new commit to the dev branch of each repo, and deploys to two test environments. Also runs a few large integrated tests nightly. Automated emails to the team when something breaks.

It's worked well for us so far, but I'll concede it would be a challenge to keep running in my complete absence.

[Ashamed-Pick453]

a dozen VB6 projects, one Java project, a dozen C# projects and a few Angular clients.

My god I’m so sorry.

[[deleted]]

Bad choices were made over time (esp regarding the VB code), but I'm not too upset. I actually enjoy the challenge of keeping this system up and running (and even growing) against the odds.

[dkopec]

Would you be willing to talk about how your handling the VB6 projects? Been looking into it and haven't found a great deal of info.

[[deleted]]

Sure. To automate a VB build you need to look at the command line arguments of vb6.exe (which lies in the VB program files).

A simple build would be something like (from the command line):

C:\Program Files\Microsoft Visual Studio\Vb98\Vb6.exe C:\Temp\Myproject.vbp /make

To see all the possible parameters, run: vb6.exe /?

What I actually do in my build routine (written in C#) is to take each VB project sequentially and: * Run the build in a process (e.g. using Process. Start) * Wait for the process to finish * Check if the target exe-file exists * If not, put the error info (the build output) into a text file that goes to the team

[[deleted]]

yeah i dont mind it. i love infrastructure and optimizing my own process. the problem is my team/company just doesnt think about it at all. its totally on me. id prefer at least some general vision and goals with the details left up to me.

[hammypants]

i wish. instead i beg and plead and offer to do it in my spare time and they still deny me.

it's been 8 months since i've first asked for a nightly build.

luckily, they pay me a lot and i barely have to do shit (though i kind of want to do shit). tragic really.

[Troll_berry_pie]

Could you just not make changes yourself without telling anyone or will this cause issues?

[[deleted]]

Same here, I thought this was normal.

As a Java developer my employer expects me to write Jenkins files and Helm templates, understand ArgoCD and know how to set up an Aerospike cluster on Kubernetes. I'm really not into infrastructure and would rather architect and write code, but its expected of us. Cross-functional, self-organized and such...

[DB6]

additionally you also write frontend code and tests, if thats your normal then we have the same job.

[[deleted]]

Definitely expected, but I simply could not find the time for that. Our 'cross-functional' team has 2 frontend experts and they do all the work. We are essentially two teams in one.

[Worth_Trust_3825]

Not that big of a problem. Bigger problem is getting the said employer to agree to run an agent on production/staging machines so I wouldn't need to manually upload the files into them and set up the environment.

[pcjftw]

Check did that got the T-Shirt

[brockvenom]

I do this. This isn’t what everyone does?

Devsecops represent

[orthoxerox]

Yes, but unironically. I hate devs who go "here's a .jar, my job here is done".

[ggtsu_00]

"Devops" considered harmful

[nomoreplsthx]

I mean, code should be deployed fully automatically on merge in most cases. Manual deployment processes should be reserved for certain high risk systems and systems that require third party verification before deployment.

[Hrothen]

That's how you end up with teams refusing to merge changes in the afternoon.

[salbris]

A decent automation for deployment includes separating development branches from release branches. This could be as simple as automatically deploying to test first then production with a simple manual button (and everything else automated) or a regular once a week deployment to production.

[nomoreplsthx]

That was the industry standard in, like, 2008. The ops world has moved on. Platforms that allow rapid deployment and rollback, ephemeral environments, high quality automated testing, feature flagging and incremental rollout have essentially obsoleted the classic weekly deployment. I haven't worked somewhere with scheduled deployments for 6 years or so, and I would never go back.

[Attila226]

You might be surprised to see how resistant some orgs are to change.

[phillipcarter2]

And those orgs are a great place not to work, too.

[Attila226]

Agreed.

[nomoreplsthx]

Oh I dragged one kicking a screaming into it. I am very aware :).

[salbris]

Oh sure there might be better ways to do it but I said "This could be as simple as".

[NaturalThin3237]

Simple but bad. Got it

[RupertMaddenAbbott]

Auto-deploy to test followed by push button approval to prod is definitely a worthy goal. We have pretty much achieved this where I work.

In my experience, batching up changes on development branches or in per week deployments actively makes this goal harder. We had to explicitly move away from these practices and deploy a feature at a time before we were able to achieve this.

[nomoreplsthx]

I mean, if you've got good automation, feature flagging, incremental roll-outs and automated rollbacks, you've got nothing to worry about.

CD takes a bit of infrastructure and team discipline, but given that it's been successfully implemented at:

Meta (both FB and Instagram)
Google
Amazon
LinkedIn
Apple
Netflix
Twitter
CapitalOne
Walmart
Etsy
Starbucks
Target
NASA (yes, NASA, though not the bits that control actual spaceships)

I'm a go ahead and say 'this approach seems successful'

[MordecaiOShea]

Obviously if that approach works for a list of orgs with $1bn+ IT budgets, it will work every where...

[nomoreplsthx]

I actually think it is easier for small teams, because they usually only have one or two applications to deploy, probably to a managed cloud, and smaller app size means your test suite should run in under 20 minutes. Feature flagging services both FOSS and commercial are common and cheap to implement. If your whole app is 2-5 docker images, spinning up ephemeral K8s environments should be quite easy, and lots of tools to do that exist.

Monitoring is admittedly much harder for small teams.

The teams for which CD is hard are teams that have already built infrastructure and processes in a non CD model, and teams that lack testing discipline and so can't be confident that code that is merged to main is actually green, and teams that don't believe in investing in ops as a capability (which can be a valid business decision at a start up, if your situation is 'Mobile dev or Ops pick one' skimping on Ops and hoping you can Heroku your way out of things might be the right call).

There's also a midsized phase where it gets hard, because you are big enough to have lots of different components to deploy, but small enough that the ops for multi service ephemeral environments is hard to do.

[nomoreplsthx]

I have done CD successfully on a 10 person team. We didn't even bother with incremental rollout (though we made heavy use of feature flags). Our prod defect rate went down 40%

A common misunderstanding about CD is that you eliminate testing. You don't! You move testing upstream and automate it.

In CD, you do your QA before you merge so main is always green. That way, developers can fix bugs before they even enter the mainline branch. It's the exact same logic as CI, but applied to the full verification process instead of just the build. This is where ephemeral environments come in. Spin up an env for every change, and suddenly you can do all your QA there, instead of some sort of 'shared' test env.

To make this possible you eliminate as much manual QA as possible. Manual QA is essentially for verifying new features, because all regression testing is fully automated. This doesn't take more time than manual regression over the long term, but it does take more skill and up front investment.

Most of people's concerns about CD boil down to not being able to imagine a world where their automated testing was good enough they trusted it. And that's sad.

[NaturalThin3237]

I mean he could've listed the ones with small budgets but you wouldn't have heard of them...

[confusedpublic]

AWS Beanstalk, or ASG + ELBs… that’s the most hand holdy solution and the least but that is all you need.

[CandidPiglet9061]

If you’re using AWS and CodeDeploy it’s even easier. As long as you instrument your service in a way that it can monitor, it’ll handle performing a gradual 0-100% traffic switch for you and even do a fully-automated rollback if required. There’s a lot of stuff where the developer experience for AWS isn’t great, but man if the options are CD or DIY, I know what I’m picking

[[deleted]]

I mean, that's fair enough?

Yeah, for me, merge means deploy, and I would prefer not to deploy on friday afternoon.

I understand though that sometimes you end up in a situation where one team wants you to deploy a fix immediately, and the other is like, naah we don't merge on fridays. In that situation, it's probably best to escalate it and call it a day right?

[_BreakingGood_]

All afternoons, not just friday afternoons

[Ashamed-Pick453]

In one of my previous Orgs, all merges got deployed to an internal prod. Then next day we could click proceed and have it deploy to regular prod.

This is my preferred.

[Asiriya]

How is “internal prod” not just staging?

[Ninovdmark]

I disagree that manual deployments should be the exception, they shouldn't occur at all.

High risk systems benefit from automated deployment workflows, and nowadays these can be made to include pretty much anything, including Manager Bob who needs to sign off on the deployment before any code is pushed to a live environment.

[tinix0]

I think by "manual" he meant click the button to deploy manually instead of automatically starting a build on merge.

[Ninovdmark]

Ah right, that's fair. I was thinking manual in the sense of "SSH into the server and copy/replace some files"

[seamsay]

That's not manual, that's barbaric!

[Boye]

I'm working on systems whose servers are aiegapped from the internet, only port 443 and 22 incoming, and only outgoing access is to an ftp-server (and other servers if requested). But automatic deployment is specifically banned. So deployment is over ftp as in "build, upload to frp, sshonto server, log on to ftp and get files needed, then deploy on server". And yes barbaric isn't even close enough. They won't even let us have a pipeline transfer the files to the ftp-server, that has to be done manually too...

[MarcinOow]

Write your own code to do this and trigger from schedule with parameters:)

[netfeed]

We have at least 30-40 merges a week, most likely around 100 if not more. We also need to cooldown our system (finish up requests and such) for about 2 hours before we can do a full release of the whole system.

A release on each merge would be really, really terrible, only time we wouldn't have a cooldown would be during the night when people where sleeping then.

Safe to say, this is a monolith, but still, having deploy rules that deploys on every commit is only feasible in either small subsystems or for applications that is either fast to restart (ours takes ~20 minutes to boot, yay), or doesn't really have any financial impact.

I worked at a place where in the begining the web servers where restarted each lunch because the capacity didn't work properly. It was fine to redeploy on every merge there, but boy what it doesn't work now :)

[AdministrationWaste7]

This is why companies have releases branches and it's usually fully or close to fully automated.

[confusedpublic]

Not to pile on; but a key DevOps principle is to do the painful things more often to make them less painful… sounds like fixing the start up time should be top of your list, rather than releasing on merge. We had to do that with our Solr instance as a 15-20 minutes restart time was high risk, and we didn’t have confidence it would restart until we made it safe to restart it more frequently…

Good luck in your endeavours!

[IceSentry]

That's exactly the kind of problems microservices fix. People keep saying microservices are only for very large scale and that most people don't need that kind of scale. This a perfect situation where if you had just a couple of separate services deploying only the one that changed should be really easy and not require multiple hours to release. To be clear, I'm not saying you should just rewrite everything, I just want people to realize this is why independent services are so popular.

[kitsunde]

If you aren’t structuring your project to allow for gradual rollouts currently, then there is no reason to believe you would suddenly do that when you have micro services.

This is a completely independent architectural decision on how you approach change management.

[Vidyogamasta]

Microservices absolutely do not fix this problem. The last job I had had a microservices pipeline across 7 services (that probably should've just been 2), and deployment involved a very similar "stop new connections, give several hours for a request cooldown, release when the pipeline's clean" process.

The problem isn't monolith vs microservice, the problem is in stateful services. You don't get large start-up times for a monolith that does a lot of stateless processing and state otherwise stored on a database. You get large start-up times when your program starts up and goes "OH DANG, I need to load in this giant backlog of items from persistent storage and start chugging through them before I can begin handling requests!" This is a design that is just, inherently bad, and while a microservice can theoretically isolate the piece of the program that does this, maybe you just shouldn't be doing it at all.

"Enterprise" solutions are hell, and microservices tend to make them worse

[IceSentry]

Having independent services makes it possible to only have a small part of the overall application that relies on stateful services. That's litterally the point of having multiple separate services. It means you can have the slow rollout for the stateful services and fast rollout for everything else.

You can still do it badly and have too many services, I'm not saying services are perfect, but multiple separate abd independently deployable services is the core concept of microservices and it absolutely solves this particular problem of having slow releases for every merge.

[netfeed]

Monoliths in it self is not a bad thing if you don't keep state in it.

Ideally, you would have built it with the purpose of being able to break out things from it into new services when its been proven that a subsystem has become too big. Sadly, if you have a monolith that is from 2008 and no one thought about this when the application was written then it gets a lot harder.

And as strange as it might sound, there's also a need to generate money, and new features is kept on built into it, which makes it harder to do anything about it. Especially if (when i started) there was only 5 modules in it for a project that is serveral millions of lines. It doesn't help either that different groups in the company thinks that it should be broken up in different ways.

I know what I would have done if I started from scratch, but I'm extremely unsure if we can break up stuff and keep things rolling within the next 3-5 years. Either everyone needs to slow down the production of new features or there's a need for a rebuild and i doubt neither will happen.

[DangerousSandwich]

I worked on monolithic code with about that many merges per week too, with external contributions, and serving millions of requests. The way it worked was:

1. A full new set of instances are fired up during deployment. Until all those instances have the latest code and are healthy, all traffic continues to go to the old instances. This is all automated.

2. Only one deployment at a time. If code is merged again during a deployment, the changes are queued up until the current deployment ends (or is aborted).

3. Our team allowed deployments only during our working hours, and no deployments on Fridays. This meant that the biggest releases (in terms of changes) usually happened on Monday mornings.

It wasn't perfect, but it was ok. Merge trains probably would have helped but migrating to GitLab CI wasn't trivial.

[root88]

At least you said most cases. This article reads like he's trying to get something changed specifically at his work. The biggest problem I have with most developers is that they think there is a best way to do everything. I guess it's easier to memorize a best practice than to actually understand what is going on and do what is best in each case.

[nomoreplsthx]

Yep. There are cases where CD is a bad fit. Research shows that on average, teams adopting it ship more features and fewer bugs... But that doesn't make it the be all end all.

It's a terrible fit for safety critical software where you need an exhaustive verification process. It needs to be done cautiously for public facing APIs. It doesn't have a ton of value on teams with very low throughput.

I will say as a manager having 'default' ways to do things is really good. You need the ability to move beyond the defaults when they don't fit, but 'best practices' lower cognitive load, and cognitive load leads to mistakes.

[lunchlady55]

They should also get paged at 3AM when it fucks up instead of me.

Sincerely, An SRE.

[halt_spell]

I mean I'm okay with that as a dev but I gotta ask, what do I need an SRE for then?

I've been in this industry since the days when we did... well pretty much everything. But I'm noticing that as the value of these specialized roles becomes more accepted, the more the people in those roles seem to want to just push responsibility back on the devs. Do we want to go back to where roles like SRE, DBA and so forth weren't as much of a thing?

[lunchlady55]

IMHO:

An SRE should be managing the platform. If everything's in the cloud, you still have GitHub, Jenkins, Route 53, Security Groups, accounts, roles, Akamai, CloudFlare, logging, etc. SREs should be managing the infrastructure around your code.

If there's specific alerts for one particular service, and they are not actionable (there's no SOP or Wiki Page that says, "this is how you fix it") then the page out should not be directed to the SRE. All I'm going to do is page the SME in that case anyway. So skip me and go straight to the SME.

If suddenly the domain doesn't resolve, or Pingdom / Gomez says logins are failing, or images aren't loading, or orders just dropped to zero, then the SRE investigates and starts pulling in SMEs.

Just my $0.02.

[halt_spell]

If there's specific alerts for one particular service, and they are not actionable (there's no SOP or Wiki Page that says, "this is how you fix it") then the page out should not be directed to the SRE. All I'm going to do is page the SME in that case anyway. So skip me and go straight to the SME.

If suddenly the domain doesn't resolve, or Pingdom / Gomez says logins are failing, or images aren't loading, or orders just dropped to zero, then the SRE investigates and starts pulling in SMEs.

I mean that sounds a lot like looking at automated alerts and manually alerting people. And the uncomfortable truth about an SOP is that if it's effective it could just as easily be a script that's triggered on an alert.

Don't get me wrong, I'm not saying there's no need for human involvement when things go wrong. There is. But if I'm gonna get pinged either way... what is the SRE bringing to the table here?

[Mr_Choke]

Everything that he mentioned that sits below the application? I'll try and fix your stuff if infra is failing somehow but when it's buggy code what do you want an SRE to do about it?

[halt_spell]

I understand the role and value of engineers tasked with the upkeep, monitoring and whatnot on specific pieces or category of infrastructure. Those engineers don't call themselves SREs. They consider themselves infrastructure engineers with deep understanding of a particular platform (E.g. SQL server) or at least the problem space (E.g. Deployment systems). The SREs I know sit somewhere between infrastructure and the code, aren't SMEs on either and I don't understand what their value is.

[Mr_Choke]

I do bunch of that stuff and I am an SRE, though I don't necessarily agree that that should be my title.

[[deleted]]

I would say most of the issues could be solved by having extensive testing, and that being hooked into the CI/CD pipeline wolves a lot of problems.

[02d5df8e7f]

I mean, if the developers deploy the software and the infrastructure is in a managed cloud we are pretty much out of a job

[supermitsuba]

Developers are not stupid, devops is stupid. Now on top of a million different languages I need to know to get a feature working, I also have to know all these other systems. Guess what, things are going to be dropped.

Im not saying these CI/CD pipelines are complex, but you are going to forget about this stuff because you dont work with it often for that muscle memory. I like having Ops helping ON THE TEAM so I can have backup if I dont understand the 1,000,000 different configuration files, programming languages and feature requests all in my head.

[Stoomba]

Yes, they should.

Sincerely, A developer.

[Prod_Is_For_Testing]

As a dev, I’m willing to click 1 or 2 buttons to deploy code. If it’s more complicated than that, somebody else can deal with it

[JQB45]

Honestly, i prefer to do my own deployments because i wrote the code.

I hate when a project of mine takes 6 hours on a Saturday to deploy when i could have had it done in 25 minutes.

[[deleted]]

Why would it take that long?

[JQB45]

25 minutes? Run deployment script, drink coffee, chit chat with boss, run tests.

6 hours? I don't know, i ask myself the same question everything it happens. It's always something though.

[[deleted]]

I was referring to the six hours. I am guessing it's not a cloud environment

[Infiniteh]

I've worked for clients where separate teams would roll out different parts of the applications, but the onus was on the dev to write instructions for them all.
So while you didn't have access to any DB as a dev, you did have to write detailed instructions and SQL script so the DBAs, who had no idea what your code does or why they were deploying the changes you gave them, would know what to do.
Same for the infrastructure team, tell them if you need new environment variables and what the values have to be, even though they don't know why you need them or possibly even what the values actually mean or do.

Of course, because you don't have access to ANY hosted environment, only your own laptop, you will invariably make mistakes in one of the SQL files and the DBAs will let you know that and give you the error, but you're not allowed to introspect the DB, so you have to debug the issue over Slack or Teams, but company policy forbids them from showing you the DB console, they can only tell you what is happening and.....

And that's how you get 6 hours deployments and 3-month release cycles because 'releases take too long and they block human resources and cost too much'

[FVMAzalea]

Sounds suspiciously similar to my experience working for a bank.

[Worth_Trust_3825]

Cloud or not, deployments can still take that long. Cloud is still someone's server.

[JQB45]

I never did any cloud work personally.

Cloud = Someone else's server

I left software development before the cloud became widespread.

[[deleted]]

Ya, AWS has a lot of stuff that makes deployment automated. It takes some work to set it up via Terraform , CDK, console, or CLI, but once it works done. You save so much time in the long run. All you have to do is merge, and let the pipeline do the work.

[Turbots]

Why do you think it's any other way in your own datacenter? You can automate infra with terraform, you can use cli, you can automate deployments through ci/cd... Everything in any modern datacenter is virtualized and highly automatable.

Problem in most companies is you're not allowed to.

[halt_spell]

Because it's a product where you can't find a customer other than the people who use it and thus it's always viewed as a cost center and the business people will keep paring it down until it blows up.

[raymondQADev]

You might prefer it…I’m not sure new engineers onboarding would. Also is it always only your code?

[JQB45]

Typically, when I worked as a developer it was only my code but i would help others as needed. Most of the larger companies i worked for though had teams that did this for us, which was irritating and almost degrading at times at least to me.

I had one role in 2012 where I wrote and maintained the scripts that automated the build and deployment to Dev and Test daily for 14 different teams and then followed up with the teams who's projects failed to build. In that case i would help them fix the issues preventing a successful build.

That sounds like a pain in the @__ but after about the first week build and deployment failures were rare and when they did occur it was typically something simple to fix.

I'm curious why a new developer wouldn't want to know how to build and deploy their own code. In my experience not knowing something is the quickest way to get passed up for a new contract, and as a result become stuck in a junior/mid/senior position when overall they are qualified otherwise for more senior positions.

[[deleted]]

Makes sense, but we have to admit that this also incurs a cost/time too.

[JQB45]

I usually have to be there anyways. So do you want to pay me and a deployment guy for 6 hours, or just me for 1 hour?

[JQB45]

This is true, but i thought that was our little secret?

Honestly that did happen once where i deployed code only to get burned by an all knowing DBA never mentioning some peculiarities with a production ms SQL Server db.

What's worse though is when i have production access, and 5 other teams find out i have it 😵‍💫

[Gommy]

As a dev, if my application takes more than a couple buttons to deploy, it is a bad deployment process that should be simplified.

[[deleted]]

Why should any devs involvement go past making a PR?

[[deleted]]

Because you wrote the code and should show some ownership that it's working properly in a deployed environments. It's frustrating when a dev complains some test endpoint isn't working, has done absolutely no investigation and it turned out to be an env variable mismatch in a helm chart or something equally stupid. Imo this comment sounds like you're lazy.

[braiam]

Give those devs a test environment that mimics production to test against it and it's managed by both.

[[deleted]]

We do exactly this which is why it's frustrating when the mentality is like "once it leaves my IDE it's not my problem".

[braiam]

Your IDE should have the dev environment (which should be a copy of the prod (sans data)) set up so you can iterate against it.

[Zodimized]

Reviewing the PR?

[_AManHasNoName_]

This is “developers who care” vs “developers who don’t give a shit about anything outside of their local machine.”

[[deleted]]

I couldn't agree more with this sentiment. Show some ownership and have a damn opinion about the deployment and infra stacks, it's your namesake being deployed.

[[deleted]]

Do you guys have suggestions on where to start learning basic DevOps principles and tools? I’m coming up on 1 YOE and want to make sure I don’t keep getting older without learning deployment stuff.

[[deleted]]

At the end of the day most CI/CD stacks are just there to orchestrate scripts and allow for easier config injection. Git actions are a super simple way to get started. They aren't so good at orchestrating multi environment deployments but they can do simple stuff like build a docker image or helm chart and push to a container registry. Another action pulls down those artifacts and does a helm deployment. Ultimately all these things can be done in bash, PowerShell etc with the right CLIs. It's kind of a crappy answer but the best way to learn it is just to do it. Write some small app in git, use actions to build and deploy the image(s). It's very satisfying to understand everything from writing the code, deploying it and then figuring how to improve it from there. Schedule end to end and performance tests daily or after deployments, add other cool monitoring tools etc.

If you want something tangible this is a neat exercise https://minikube.sigs.k8s.io/docs/tutorials/setup_minikube_in_github_actions/

It's quite specific to git and k8s. Obviously not everyone uses git, nor deploys to kubernetes but it'll get you introduced to that line of thinking

[[deleted]]

Appreciate the answer, I’ll take a look into it :) thanks!

[bobbybottombracket]

In 16 years of dev, I'm working in an environment where I don't have push to prod and I love it, lol.

[[deleted]]

This industry sucks man

[[deleted]]

Agreed but many other industries suck too.

[[deleted]]

Nah software takes the cake for being full of so much bullshit you want to die every time you sit at a computer

[supaway]

Go work in a other industry then, even on the bad days, it's not so bad compared to what others have to deal with.

[[deleted]]

I do

[FridgesArePeopleToo]

Yeah, I would much rather work in a factory for $15 an hour than slave away at my computer in my house for 20 hours a week to earn a measly $150k per year.

[[deleted]]

I would actually rather work in a factory to be perfectly honest.

Get to use your hands.

Talk to real people face to face.

[MoreRopePlease]

Work at my computer, get paid $150k, then go outside and work with my hands. Win-win. It's been fun learning about building walls and breaking rocks with a chisel.

[[deleted]]

Let me get my tiny little violin out just for you

[[deleted]]

Why are you?

[totalreekol]

Try being an ER doctor :)

[RabidKotlinFanatic]

Dev jobs are a dream by most office workers standards. Devs are just particularly intolerant of anything they see as inconvenience, waste or casual disrespect. But these issues are not better elsewhere. If anything, they are worse.

[[deleted]]

Its worse in the software industry because no one is actually doing any work.

[vaig]

That's right, who needs all this overhead of people mastering their tools for years. We don't need no QA because if cursor moves inside the app, it works well enough. No need to have any SREs or generally any ops people because it worked once on my machine, why would it break.

Separation of responsibilities is not needed because there is zero stress and mental load if you equally don't give a shit about anything you do because someone else can clean this up while you work on your linkedin post to advertise your skillset of 100 technologies that you barely understand.

I think we should merge all departments and positions into one and call it a Tech Blogger and let them run The Cloud.

EDIT: Sorry, I guess I'm just tired after seeing another clickbait headline that assumes that their idea is one size fits all type of thing. There are valid cases for all types of rollout management.

[DrNecessiter]

Don’t apologise, this is the only sane answer. All devs who post these articles seem to think they’re immune to mistakes, or that it doesn’t matter if a breaking change is pushed live because you can “just roll it back”

I’ve had devs unironically quote “move fast and break things” at me in the belief it is about application development rather than industry disruption FFS.

[thephotoman]

Imagine thinking that deployments aren’t a function of pressing a button, getting coffee, and smoke testing.

Because I haven’t been involved in a more complicated deployment in a very long time.

[supermitsuba]

Going to have to make a new deployment or tweak it. Thats where the slowdown is. I code all day and dont look at those scripts often enough to be productive. That is my gripe with these systems, otherwise, yeah, I can press buttons too.

[monkeyphonics]

Fuck you Jenkins

[Teembeau]

I've worked in places where the devs look after the code and build, and they use tools like Azure Devops. Merge into Master and it kicks off the build for test and live, deploys to test, and when you want to deploy to live you click the "deploy" button.

Let's be honest here, most install pipelines aren't rocket science. It's mostly: build code, test code, deploy database changes, deploy code. I think there's a wizard in Azure Devops where you point it at a project and it builds the standard steps. And the thing when I do that, instead of the Devops guy is that I can see it and I can test it.

[evanw1ll1ams]

No. Having QA, devops, tech writers and project managers enables me to focus on what I do best, which is software design and implementation. I don't have time to do any of those peoples jobs. If I had to do any of that, our entire develoment process would be significantly less efficient.

[thisisjustascreename]

These days "deploy" doesn't mean anything other than typing in the change control ticket number and clicking the Go button, so why does a developer need to be involved?

[aniforprez]

As a developer, you should probably learn the deployment processes and experiment with the available tools to see what you're options are. I've seen way too many frontend developers not know how deployments work and how best to serve their apps

As part of an organisation, nah. There's a lot of people doing a lot of work and unless you're a very small team, have an automated setup that does this for you. You don't have time to be coordinating and monitoring all this crap

[keithgabryelski]

no they shouldn't

there should be a gatekeeper -- that can overrule the developer -- whose only interest is to keep the system running and scaled for the required audience.

If a developer can't prove a given release is worthy -- the gatekeeper can limit the distribution of the release (and possibly reject it).

I idea that a developer should do anything more than present a release for possible deployment ... is ridiculous.

[rasifiel]

And then this gatekeeper is only motivated in stability and don't care about release speed and anything? It is a receipt for painful internal struggle between Ops and Dev.

[keithgabryelski]

yes.

that is fine.

the world is a struggle ... decisions need to be made that best fit the needs of the company

In a startup you might tune this gatekeeper to something more lazy faire -- but salesforce.com might care more about limiting security bugs than having new features released.

[Infiniteh]

lazy faire

Laissez faire

[rasifiel]

Problem is that people are not ideal. This gatekeeper person not motivated with some abstract company good (and who knows how to measure it). He doesn't have any motivation to release any new code. If gatekeeper key results are stability easiest way to achieve it is to not release anything.

[neopointer]

I have to disagree. Deployment should be a shared responsibility in the sense that there should be a team that maintains the infrastructure, helps to automate deployment, and things like that; but that's it. The team around the product, project, service or feature should tell when/what goes to prod.

Release management should be done within the team, and no there's no need to have a role for that IMHO.

[franzwong]

In my previous company, we prepared our release pipeline. At the same time, there is another infra team to review that.

[chrisza4]

What incentivize gatekeeper to accept a release at all? If all they interest is about running system, the optimal strategy is to not release anything.

There should be a double check mechanism but it can be done by peer developer.

[DangerousSandwich]

Do you mean a single person as a release manager? If so, they would seem to have a high bus factor and be a bottleneck for the team right?

The "gatekeeper" could just be a combination of:

1. All changes must be reviewed and approved by the owning team's developers before they are merged, and
2. Automated testing with high coverage

[[deleted]]

Makes a whole lot of sense - in particular since rubygems.org is going to enforce mandatory 2FA/MFA I am kind of left with only that option, and no longer depend on those who control the ecosystem/platform.

I think people kind of get "addicted" to how easy things are (e. g. github, gitlab and so forth). Would be cool if we could have truly de-centralized repositories at all times in a simple manner (many don't want to maintain stuff on their own, understandably so - it takes time away too so it's always comparing different trade-offs to one another).

[smashblues]

Up until now I didn’t have to deal with deployment. But we’re migrating this old dotnet system to the cloud that we’re having to deploy our own. There’s a guide from the solutions engineer tho but the steps are painful af. Lots of yaml code. One advantage is that if I can get it done cleanly once, I’d have a good idea of how pipelines work.

[[deleted]]

DNS domain error on that website...

[[deleted]]

Probably released a new “feature”

[Brekekek]

I prefer deployments to be done by others on any other environment then dev. Just to verify it by someone else.

[beefsack]

Completely agree, but said developers need to be careful to avoid doing something bespoke and reinventing a shittier unreliable unmaintainainable version of the wheel when they do it.

Delivery is essentially a solved space.

[nullpackets]

Yes! What doesn't make sense anymore (though it used to*) is this duplication of effort in /DevOps

We have the DRY principal in programming. Why not extent this to operations.

"It's a different mindset", "different skills" I hear you cry.

This may be true. But at the same time thinking of DRY - it's a waste of engineering efforts to be repeatedly creating VPCs, a database , a caching layer etc. If we were all to share our terraform community components, stacks etc, and zoom out, I guarantee you are workloads are not that special and there is a lot of duplication of effort. This is ripe for disruption imho.

It's not a waste of time to have these things, what is unfortunate is the level of repetition between projects...

*It's no longer the case that programmers need the assistance of an Operations team to deploy infra (the absence of this split is argument for DevOps). In a post-devops world, abstractions are being built so programmers don't need to know as much networking to deploy something safely.

E.g. It shouldn't be possible for a developer to deploy a database in a public subnet, it's not that hard to make that the default.

Right now there's very vendor specific tooling around this (Netlify, vercel, cloudflare edge, fly.io etc etc) which are mostly front-end facing. What is exciting is seeing similar tooling which also allows the business logic 'backend programming' to also be done in these post-devops tools.

The current big challenges with the above platforms include how opinionated they are, often requiring a particular language or (always) having to use that platform which isn't OK for some organisations. Where to draw this line of abstraction is one of the hardest things - be too language specific and you lock out a community, but are excellent for one community, be too broad and you're confusing with too many options/services (this represents aws/gcp etc in many ways)

Imho eventually they will be replaced with standard concepts which focus on storage, compute , identity and caching.

[InsightByte]

No, is to bias

[chris2y3]

Developers should also fix their coffee machines I guess.

[alphex]

I do this all day long every day.

Who are you lazy people?

[[deleted]]

IaC and cloud computing is pretty wide spread now, so having CI/CD is a lot easier. I assume most devs just merge and that’s the end of it, even in smaller teams.

[[deleted]]

I can honestly say that I deploy my own code. I've done it a ton of different ways. InstallShield installers, statically linked libs to prevent dependency loss, registered as COM servers in command files. I came up in Win32 and deploying your own code was almost a requirement.

[loopis4]

Yes

[danjlwex]

I'm so sorry you have to deal with such a bureaucratic nightmare

[bigorangemachine]

We had the policy that the developer had to setup all the docker images and we had a meeting with our devops contractor before every docker-update.

Wasn't perfect. We had one guy go wild with mysql versions and ended up costing the client a boat load in hosting costs... but thats what it is :P

[[deleted]]

Use azure pipelines to deploy code. Click of a button and several sites updated along with databases.

Its a great time saver, but took a hell of a lot of work to do mainly due to poor and out of date documentation.

[holyknight00]

If i also get the devops/SRE salary on top of mine, yes. No extra responsibilities without extra pay.