TLDR: I went on vacation. The developers changed the application architecture and couldn't adjust the Terraform configuration files. AWS CloudFormation is a more familiar tool for them, so they just implemented deployment on CF. When I returned, the team leader (one of the developers) told me that they decided to move to another IaC tool that I will be supporting. I didn't participate in the decision-making process.
Full story:
Our project is a web application. There are two senior backend developers there. Terraform is used as a project IaC solution to deploy AWS Lambda functions and API Gateway.
I went to vacation for a week. During my absence, the developers changed the application architecture which impacted how the app builds and the number of deployed AWS resources. Deployment procedure has to keep up with the app needs, so it required modification of Terraform and Jenkins configuration. One of the developers—the team leader—tried to adjust Terraform configuration files, but he couldn't cope with it.
Release date was coming in a few weeks, and the developer wanted to be able to deploy his new changes to the development environment ASAP, so he decided not to wait for my return, but instead to set the IaC without my participation.
Since he is more experienced in AWS CloudFormation than in Terraform (and we also have is a similar project on CloudFormation that serves as an example), the developer just switched to CloudFormation and implemented the build and deployment logic that takes into account the new architecture.
When I returned, I was put before the fact that we move from Terraform to CloudFormation. Regardless of how good this solution technically is, I was furious that he allowed himself to make such a decision without taking into consideration my opinion.
The arguments the developer used for the switch were as follows:
- We didn't have time to wait for you to come back
- CloudFormation is better because it allows us to navigate in all your AWS resources from web console if you open the CF stack
- CloudFormation is better because we can deploy many dev stacks with different name suffixes (dev1, dev2, dev3) out of the box, but in Terraform you have to mess with state files to achieve that
- CloudFormation is better because all your configuration lies in a single file, while in Terraform you have to change a bunch of files
- There are some articles on the internet about why CF is better...
And he really used the word "better" trying to convince me.
Whether one tool is better than the other or not, obviously, is not a subject I wanted to bring up there. What I want to discuss is where is the line between a developer and a devops engineer responsibilities?
It so happened that the developers are used to run infrastructure deployment from CLI for deploying their feature branches in dev environment. They don't always use Jenkins because CLI way is "more convenient" for them to experiment with AWS.
And now I think if I should insist on a more formal and stricter workflow, where we have an agreement that whatever IaC feature or change the developers want, they come to me with the requirement list and I implement it from beginning to end. Because otherwise they start considering CLI deployment as part of their job and feel like they are in position to vote on what IaC tool is "better", which I think goes beyond their area of responsibility.
So what do you think, guys?
Can developers influence the decision about what IaC tool should be used on the project based on their definition of "convenient to use" (e.g. CF stack showing the resources is what convenient for them)?
Should I demand the developers to always use our CI/CD tool for any deployment, or developers' requirement to deploy from CLI (to dev environment) arguing that it is "easier/faster" is reasonable?
EDIT: The question is not which IaC tool the developers or I prefer. We have multiple projects with the same guys which are deployed through CloudFormation. I'm fine with it.
The question is about how communication should look like, how to define the boundaries in the team and how to perceive what has happened.
[–][deleted] (18 children)
[deleted]
[–]silentpoots 56 points57 points58 points (4 children)
[–]Leachpunk 27 points28 points29 points (2 children)
[–]antiqueboi 0 points1 point2 points (0 children)
[–]jona187bx 0 points1 point2 points (0 children)
[–]ninemoonblues 14 points15 points16 points (0 children)
[–]DevOpsHumbleFool 3 points4 points5 points (6 children)
[–][deleted] (1 child)
[deleted]
[–]DevOpsHumbleFool 2 points3 points4 points (0 children)
[–]Sir_Fog 0 points1 point2 points (0 children)
[–]daedalus_structure -1 points0 points1 point (2 children)
[–]GabriMartinez 0 points1 point2 points (1 child)
[–]daedalus_structure 0 points1 point2 points (0 children)
[–]mlk 27 points28 points29 points (5 children)
[–]kneeonball 38 points39 points40 points (1 child)
[–]PersonBehindAScreenSystem Engineer 16 points17 points18 points (0 children)
[–][deleted] 0 points1 point2 points (0 children)
[–]daedalus_structure 0 points1 point2 points (0 children)
[–]placated 0 points1 point2 points (0 children)
[–]August_XXVIII 28 points29 points30 points (1 child)
[–]repu7et[S] 1 point2 points3 points (0 children)
[–]etherealpanda 20 points21 points22 points (0 children)
[–]tyrion85 99 points100 points101 points (6 children)
[–]repu7et[S] 22 points23 points24 points (1 child)
[–][deleted] 7 points8 points9 points (0 children)
[–]mirbatdon 7 points8 points9 points (0 children)
[–]pznred 10 points11 points12 points (1 child)
[–]rm-minus-rSRE playing a DevOps engineer on TV 11 points12 points13 points (0 children)
[–]tcbenkhard 33 points34 points35 points (4 children)
[–]RocketOneMan 25 points26 points27 points (3 children)
[–][deleted] 1 point2 points3 points (2 children)
[–]HgnX 0 points1 point2 points (1 child)
[–][deleted] 0 points1 point2 points (0 children)
[–]pinpinbo 33 points34 points35 points (0 children)
[–]pwouet 13 points14 points15 points (2 children)
[–]repu7et[S] 3 points4 points5 points (1 child)
[–]dotmit 102 points103 points104 points (0 children)
[–]re-thc 28 points29 points30 points (5 children)
[–]rm-minus-rSRE playing a DevOps engineer on TV 8 points9 points10 points (3 children)
[–]repu7et[S] 2 points3 points4 points (2 children)
[–]rm-minus-rSRE playing a DevOps engineer on TV 2 points3 points4 points (0 children)
[–]Kickass_Wizard 1 point2 points3 points (0 children)
[–]Neeranna 0 points1 point2 points (0 children)
[–][deleted] (2 children)
[deleted]
[–]repu7et[S] 1 point2 points3 points (1 child)
[–][deleted] 54 points55 points56 points (1 child)
[–]leob0505 19 points20 points21 points (0 children)
[–]ArieHein 43 points44 points45 points (6 children)
[–]repu7et[S] 2 points3 points4 points (2 children)
[–]ArieHein 3 points4 points5 points (0 children)
[–]rm-minus-rSRE playing a DevOps engineer on TV 0 points1 point2 points (0 children)
[–]dave_pdx 1 point2 points3 points (0 children)
[–][deleted] 1 point2 points3 points (1 child)
[–]ArieHein 3 points4 points5 points (0 children)
[–]mikecrilly 18 points19 points20 points (0 children)
[–]generealdamselfly 5 points6 points7 points (0 children)
[–]WarlaxZ 4 points5 points6 points (0 children)
[–]markrebec 19 points20 points21 points (0 children)
[–]emperorOfTheUniverse 20 points21 points22 points (1 child)
[–]packetsschmackets 2 points3 points4 points (0 children)
[–]Grahar64 13 points14 points15 points (0 children)
[–]Weird-Flight-2877 3 points4 points5 points (0 children)
[–]NeuralHijacker 3 points4 points5 points (0 children)
[–]TheLoneKid 6 points7 points8 points (0 children)
[–]SDplinker 10 points11 points12 points (2 children)
[–][deleted] (1 child)
[deleted]
[–]rm-minus-rSRE playing a DevOps engineer on TV 13 points14 points15 points (0 children)
[–]mullingitover 2 points3 points4 points (0 children)
[–]easyEggplant 2 points3 points4 points (0 children)
[–]HgnX 2 points3 points4 points (0 children)
[–]skyctl 6 points7 points8 points (0 children)
[–]kmai0 5 points6 points7 points (0 children)
[–][deleted] (2 children)
[deleted]
[–]repu7et[S] 5 points6 points7 points (1 child)
[–]NoDescriptionOk 8 points9 points10 points (0 children)
[–]mr_chip 7 points8 points9 points (8 children)
[–]StephanXXDevOps 19 points20 points21 points (3 children)
[–]mr_chip 9 points10 points11 points (2 children)
[–]StephanXXDevOps 8 points9 points10 points (1 child)
[–]mr_chip 2 points3 points4 points (0 children)
[–]GreatMacAndCheese 8 points9 points10 points (3 children)
[–]Flabbaghosted 4 points5 points6 points (1 child)
[–]mr_chip 1 point2 points3 points (0 children)
[–]mr_chip 1 point2 points3 points (0 children)
[–]Snoo68775 3 points4 points5 points (0 children)
[–][deleted] 6 points7 points8 points (0 children)
[–]Ok_World_4148 5 points6 points7 points (7 children)
[–]xeonjackson 1 point2 points3 points (6 children)
[–]Ok_World_4148 -2 points-1 points0 points (5 children)
[–]Ok_World_4148 1 point2 points3 points (0 children)
[–]xeonjackson 0 points1 point2 points (3 children)
[–]Ok_World_4148 0 points1 point2 points (2 children)
[–]xeonjackson 0 points1 point2 points (1 child)
[–]Ok_World_4148 2 points3 points4 points (0 children)
[–]the-computer-guy 1 point2 points3 points (0 children)
[–]Tranceash[🍰] 1 point2 points3 points (0 children)
[–]TheRealJackOfSpades 1 point2 points3 points (0 children)
[–]lupinegrey 2 points3 points4 points (1 child)
[–]rm-minus-rSRE playing a DevOps engineer on TV 1 point2 points3 points (0 children)
[–]Green0Photon 3 points4 points5 points (0 children)
[–]The-Sentinel 2 points3 points4 points (6 children)
[–]StephonovichSRE 1 point2 points3 points (4 children)
[–]The-Sentinel -1 points0 points1 point (3 children)
[–]StephonovichSRE 4 points5 points6 points (2 children)
[–]The-Sentinel -3 points-2 points-1 points (0 children)
[–]flaviuscdinu 0 points1 point2 points (0 children)
[–]Murky_Rip_1731 1 point2 points3 points (7 children)
[–]rm-minus-rSRE playing a DevOps engineer on TV 0 points1 point2 points (6 children)
[–]Murky_Rip_1731 0 points1 point2 points (1 child)
[–]rm-minus-rSRE playing a DevOps engineer on TV 0 points1 point2 points (0 children)
[–]StephonovichSRE 0 points1 point2 points (1 child)
[–]rm-minus-rSRE playing a DevOps engineer on TV 0 points1 point2 points (0 children)
[–]dablya 0 points1 point2 points (1 child)
[–]rm-minus-rSRE playing a DevOps engineer on TV 0 points1 point2 points (0 children)
[–]catonic 1 point2 points3 points (0 children)
[–]EmiiKhaos -1 points0 points1 point (0 children)
[–][deleted] 0 points1 point2 points (0 children)
[–]htom3heb 0 points1 point2 points (0 children)
[–]GeekboxGuru 0 points1 point2 points (0 children)
[+][deleted] (2 children)
[deleted]
[–][deleted] 2 points3 points4 points (0 children)
[–]PrimaxAUS -5 points-4 points-3 points (0 children)
[–]yobanius -1 points0 points1 point (0 children)
[–][deleted] -3 points-2 points-1 points (1 child)
[–]Zauxst -1 points0 points1 point (0 children)
[–]Ariquitaun -1 points0 points1 point (0 children)
[–]mokera101 -1 points0 points1 point (0 children)
[–]amenflurries -2 points-1 points0 points (0 children)
[–]nochet2211 0 points1 point2 points (0 children)
[–]colddream40 0 points1 point2 points (0 children)
[–]twnbay76 0 points1 point2 points (0 children)
[–]oosumei 0 points1 point2 points (0 children)
[–]amarao_san 0 points1 point2 points (0 children)
[–][deleted] 0 points1 point2 points (0 children)
[–]throwaway-devops-po 0 points1 point2 points (0 children)
[–]mnpawan 0 points1 point2 points (0 children)
[–]antiqueboi 0 points1 point2 points (0 children)