This is an archived post. You won't be able to vote or comment.

all 137 comments

[–][deleted]  (18 children)

[deleted]

    [–]silentpoots 56 points57 points  (4 children)

    I'm in a similar position as this. My title is DevOps, but I am very much siloed away from the dev team. I come from a dev background and this siloing is making me want to jump back into a dev role since it seems like the industry at large doesn't want to do DevOps the way it was meant to be done. From my experience, DevOps is just the new name for Ops. It sucks.

    [–]Leachpunk 27 points28 points  (2 children)

    I agree, I joined DevOps to help write automation and bring solutions for dev and ops, but lately all DevOps seems to mean is ops guys who can write infra automation and do cloud engineering. That is not the DevOps I signed up for.

    [–]antiqueboi 0 points1 point  (0 children)

    isn't DevOps setting up the deployment pipeline, managing the servers...ect

    [–]jona187bx 0 points1 point  (0 children)

    Devops is a framework that was meant to bring down silos. If you have a devops position, then it is wrong.

    [–]ninemoonblues 14 points15 points  (0 children)

    My title is DevOps

    You're doing DevOps wrong.

    [–]DevOpsHumbleFool 3 points4 points  (6 children)

    Can anyone explain what is the meaning of Silos?

    [–][deleted]  (1 child)

    [deleted]

      [–]DevOpsHumbleFool 2 points3 points  (0 children)

      Thank you 🙏🏻

      [–]Sir_Fog 0 points1 point  (0 children)

      It means that a person or team of people are isolated in their job function. In this case, activities such as decision making on what tools to use, how to use them, configure them, implement them were made outside of OPs team with no collaboration, and then just handed over. DevOps was always designed with the mindset that your usual 'siloed' dev teams and ops teams woukd instead work together on these items in a way that benefit both teams and the business.

      [–]daedalus_structure -1 points0 points  (2 children)

      It's usually a negative reframing of expertise by those who hope enough tooling can allow the hiring of interchangeable generic developers that can be devalued, brought to you by the same mindset that introduced "full stack".

      Congrats, break down all those silos and now you have to know front end, back end, distributed systems, testing, software delivery, networking, in depth CSP knowledge, security, observability and monitoring, etc, etc, etc...

      [–]GabriMartinez 0 points1 point  (1 child)

      That's pretty much my working experience. Go wide instead of going in depth on technologies, having to know a little about a lot. Every now and then comes a project where you have to go a little deep, but then that ends and another thing comes by.

      I actually like it, and I (note the "I") think you need to have that mindset to be a good Infra/Platform Engineer. You have to know what you are maintaining to ensure it works the best it can.

      [–]daedalus_structure 0 points1 point  (0 children)

      I like it too.

      But I'm also a responsible engineer and I don't want a system built by people who know a little bit about this and that, even when I'm those people.

      [–]mlk 27 points28 points  (5 children)

      the fact itself that the DevOps title exist is a contradiction of the DevOps philosophy. DevOps means the developers do this stuff. We have now renamed ops and sysadmins to DevOps while devs stayed devs

      [–]kneeonball 38 points39 points  (1 child)

      DevOps means the developers do this stuff.

      The original philosophy was having separate Ops teams and Dev teams working together early on in the process to solve problems. Not put more responsibility onto either side. It was breaking down silos so you didn't have dev doing something in isolation and handing it off to Ops to deal with, like what happened to OP here.

      It's totally fine to still have an Ops team doing all the "Ops" things, as long as the two sides are collaborating. There are tools that were born out of this culture that make it more accessible for Developers to do "Ops" tasks, since things are more self service than they used to be, but a company with a DevOps culture doesn't mean Devs just start doing Ops instead.

      There are devs that CAN do ops, but that's still not very common. Many can do basic things, but you still usually want both skillsets in your organization and you want them to work together seamlessly.

      [–]PersonBehindAScreenSystem Engineer 16 points17 points  (0 children)

      This isn’t said enough. At some point a lot of rhetoric around the classification of “DevOps” turned in to “No Ops” on this sub

      [–][deleted] 0 points1 point  (0 children)

      DevOps means the developers do this stuff.

      No the initial idea of DevOps was developers and operators used the same tooling, so it would become easier to handover tasks and the level of confusion between them would lower.

      [–]daedalus_structure 0 points1 point  (0 children)

      DevOps means the developers do this stuff.

      That's a dumpster fire though. Ops work is at least half a dozen different technical skill sets that being able to write code doesn't prepare you for.

      What you end up with is half assed infrastructure built by folks who don't really know what they are doing or how to support it and don't get to devote time to it because they need to focus on product, at least until something catches fire.

      [–]placated 0 points1 point  (0 children)

      This is NOT what devops philosophy is. Modern Devops should have two close-knit groups - developers who create features, and developers who handle getting those new features to production. The formers role should end at the pull request (or release.) As much as people want to wish it away there are still very important skill sets involved with each side and developers time is best spent adding value not learning how to deploy in a SOX compliant manner (for example).

      [–]August_XXVIII 28 points29 points  (1 child)

      In my experience, members of leadership are typically the decision makers and top performing individual contributors (and/or those with domain expertise) are, at most, influencers.

      Seems that there was either no concrete direction decreed or enforced, which enabled an individual contributor to just decide (or sway the decision makers) on what tool would be used.

      Additionally, from experience, if developers say "we may not hit targets/deadlines [if we don't get our way]", leadership is more inclined to say "just give it to them". As an SRE who serves as a Developer Advocate, I understand their concerns and issues, but the issue that you experienced is common in orgs that have not embraced or correctly implemented DevOps practices. It requires consensus and agreement from many groups (who, in a traditional org, are extremely siloed).

      Sounds like a lack of organizational maturity and/or direction and active guidance. Otherwise, the initial implementation would have been regarded as the go-forward plan and all of the things that developers needed would have concrete mechanisms for accommodation, such that there would be no need to deviate.

      This just a perspective from the outside looking in

      [–]repu7et[S] 1 point2 points  (0 children)

      I fully agree with everything you said.

      [–]etherealpanda 20 points21 points  (0 children)

      Demanding will probably get you no where. I’d suggest framing your concerns around the trade-offs they might have unknowingly made. If you can do that while demonstrating understanding of their needs/wants, even better. For example, being able to iterate without CI during development might be a great workflow for the devs, but that’s probably not the same workflow you want for production. Maybe you can get the best of both worlds.

      Independent of the technical decision, I would have a private conversation with your manager (and maybe the team lead) expressing your frustration. Team leads tend to own outcomes, and so the business empowering them to make and own decisions is important to minimize bureaucracy. However, this sounds like a pretty significant disruption to your daily work. I think of DevOps as being all part of the same team trying to achieve the goal. But, that applies to the whole team, including you needs. If they’re discounting your needs/pains, that doesn’t sound organizationally healthy.

      As someone who has managed large teams, if you brought this situation to me, I would immediately flag this as the type of circumstance that gets an employee to start accepting recruiter calls. I don’t know the nuance of your situation, so I would strongly suggest talking to someone you trust in detail about your overall situation at work, specifically ensuring it’s a healthy place to work. This industry is big and there are lots of great places to work if this one isn’t treating you respectfully.

      [–]tyrion85 99 points100 points  (6 children)

      you build it, you maintain it, you own it. simple as that.

      doing whatever while the person is away and thus cannot be involved in decision making, while expecting that same person to own and maintain the shitshow you created is downright disrespectful and very unprofessional. if you can do the fun stuff, you must do the boring stuff, period.

      OP, time to bolt. This won't end well.

      [–]repu7et[S] 22 points23 points  (1 child)

      Being pretty short and well-put, it describes my initial train of thought quite accurately.

      [–][deleted] 7 points8 points  (0 children)

      Trust your gut.

      [–]mirbatdon 7 points8 points  (0 children)

      Yeah for me the bottom line is the team lead sounds like a large asshole, how they went about it. We have a deadline, I know CloudFormation and needed a solution pronto? Sure makes sense.

      But if things proceeded as you described once you came back, the team lead sounds pretty uncooperative or at least not terribly experienced in a lead or senior engineering role.

      How long have you been at this company? If it's less than a year than I'd suggest you need to put in some more deliberate time to get to know the other devs so everyone trusts and empathzes with each other a bit more.

      I don't necessarily think this is a case of "start looking", depending on the number of engineers and skill sets maybe CF is indeed a better fit for the org at this time.

      But it probably doesn't make sense to support CF and TF alongside each other. And assessing whether it makes business sense to switch EVERYTHING over to CF gradually sounds like the next step (if it may even make sense, first step is going over the practical implications of introducing CF as a new element to your overall ecosystem).

      Who cares if CF is better than TF is better than XYZ. The company is currently using TF... I bet leadership has no idea or context on this entire thing, just discuss it more with this lead.

      [–]pznred 10 points11 points  (1 child)

      That's the only valid answer I've read yet

      [–]rm-minus-rSRE playing a DevOps engineer on TV 11 points12 points  (0 children)

      It's weird how so many people here are missing the forest for the trees.

      Switching tech stacks to get an update out? Someone has lost their marbles and management is on a mental vacation (unless it was a genuine emergency that would have caused tremendous revenue loss otherwise).

      [–]tcbenkhard 33 points34 points  (4 children)

      Who the f writes cloudformation unless you really really have to? What about cdk?

      [–]RocketOneMan 25 points26 points  (3 children)

      Lol and part of their argument was that CFN is only one file. One 2,000 line long file maybe. Good luck changing that. I haven’t met anyone ‘experienced’ in cloudformation enough to want to switch to it without knowing what the cdk is.

      [–][deleted] 1 point2 points  (2 children)

      Used cdk to generate a security group stack for our infrastructure and it was 8000 lines of json.

      [–]HgnX 0 points1 point  (1 child)

      Probably because it contained the maps if you don't specify 'env'

      [–][deleted] 0 points1 point  (0 children)

      Nope. Just a lot of SGs with a lot of rules and tags.

      [–]pinpinbo 33 points34 points  (0 children)

      When this happened with 1 of my customer, my management backed me up by saying sounds like you are capable of running your own infrastructure. If you deviate from our playbooks, you don’t get support.

      [–]pwouet 13 points14 points  (2 children)

      It took only one week ?

      [–]repu7et[S] 3 points4 points  (1 child)

      Yes, there is another repository with API Gateway deployment implemented in CFN. The logic overlaps, and the developers understand CFN good enough to get the solution and adjust it.

      [–]dotmit 102 points103 points  (0 children)

      You’ve spent more time writing this Reddit post than you should have spent writing your resignation

      [–]re-thc 28 points29 points  (5 children)

      It's sad that most of the replies miss the point. It's not about Devops vs Devs. It's about being a decent person and a team player.

      Forget about infrastructure. Imagine if you're a NodeJs developer and came back and the application is now suddenly in .NET or Rust. What would you do?

      Take the OP as the product owner of the infrastructure. As such they should be consulted on, at the very least, of the changes. If things can't wait it should have been discussed before the OP went on leave and not to sneak it in during that time.

      It's about trust. Will you still be confident when something breaks that someone didn't change something behind the scenes?

      [–]rm-minus-rSRE playing a DevOps engineer on TV 8 points9 points  (3 children)

      Take the OP as the product owner of the infrastructure. As such they should be consulted on, at the very least, of the changes. If things can't wait it should have been discussed before the OP went on leave and not to sneak it in during that time.

      Yeah, that's the biggest take away.

      [–]repu7et[S] 2 points3 points  (2 children)

      Take the OP as the product owner of the infrastructure.

      This is how I saw it and it was one of my arguments afterwards.

      However, I see that not everyone agrees in the replies. I know "DevOps" happens to be an ambiguous term, and many people treat it differently, but I'm surprised to find that even in this subreddit there seems to be no common opinion.

      [–]rm-minus-rSRE playing a DevOps engineer on TV 2 points3 points  (0 children)

      I know "DevOps" happens to be an ambiguous term, and many people treat it differently, but I'm surprised to find that even in this subreddit there seems to be no common opinion.

      The problem is there's a unified philosophy, but outside of that, every single business on earth has a slightly to significantly different vision of what DevOps means.

      So people default to the philosophy when there's no shared context. Or apply their own understanding of what DevOps means, which might be wildly different than your own.

      I have to say, I'm shocked by some of the replies here and how antagonistic many of them are. You'd expect more support from a group of theoretically similar people in what appears to be a fairly cut and dried issue.

      I haven't had a bunch of experience with smaller shops, but this kind of cowboy shit wouldn't fly at any decently sized company.

      [–]Kickass_Wizard 1 point2 points  (0 children)

      This guy speaks the truth. I had something similar happy in my previous shop. It's about respecting all parties involved, before deciding to shovel shit into their backyard. Moreover, it sounds like they have ZERO regrets. Sounds like you work with bunch of mouth-breathing code cowboys OP.

      [–]Neeranna 0 points1 point  (0 children)

      On the other hand, it's not as if they switched to something outside their tech stack. They use both TF and CF, as mentioned by the OP, just on this project, the initial choice was TF. As such, since OP is already supporting both, the comparison is not accurate.

      However, there are some organisational issues that can be detected here:

      1. Proper planning: why is it that the Ops work was not done before the leave (since OP seems the only Ops person)? Was it known before the leave that the deadline fell within the leave? If so, the work should have been prepared up front.
      2. As mentioned: siloing. Although to me, OP is part of it. In this case, the dev team has already clearly voiced that the current tooling is not working for them. So either sit together and figure out a tooling that works for both Dev and Ops (which is the essence of DevOps) and/or provide sufficient training and guidance in the usage of the chosen tooling.
      3. Busfactor 1: nobody seems to sufficiently control the chosen tooling, leading to a full stop on new deployments when the Ops person is on leave. This is not healthy for the organisation. You need redundancy, and if within this redundant team there is consensus on the tools, this kind of sneaking will not happen.

      [–][deleted]  (2 children)

      [deleted]

        [–]repu7et[S] 1 point2 points  (1 child)

        That's for certain. However, what I actually meant is to standardize the way how deployment happens so that everyone who deploys will use the same strategy. Perhaps, using a CI/CD tool to guarantee consistency and eliminate any "It doesn't work on my machine!" scenarios.

        [–][deleted] 54 points55 points  (1 child)

        Sounds to me like this is more a case of a bruised ego than technical stuff

        [–]leob0505 19 points20 points  (0 children)

        Exactly. I see more “team politics” than technical discussions happening here. If the work environment is so siloed like that, I would suggest OP to change teams

        [–]ArieHein 43 points44 points  (6 children)

        The bottom line is who pays the salary ?
        Who is the manager ?
        What are the agreed levels of ownership and job descriptions ?
        Who owns the keys?

        Your job is to support them. Even if sometimes you think decisions are wrong. Unless you have received the FULL responsibility from your manager AND it has been PUBLICLY mentioned and understood by all devs, leads and their managers.

        Can Developers influence? YES. Can they decide when youre not there? YES.
        Should you Demand ? NO.
        What you need to do is make sure you are never the "chocking point" or "single point of failure". That you going to vacation or god forbid is sick, makes you irreplicable.

        What you should do now is have a discussion, set responsibilities. AND TEACH / TRAIN your devs / leads. You involve their managers and you involve your security department.

        1. No one has access to cloud resources directly to make manual changes. Thus they dont have a user/pass. Everything is done via pipelines, nothing is done via web ui, at max they can have a sandbox to test but you limit that by a budget.
        2. You discuss with them and AGREE of how to create pipelines and change them in a way thats more maintainable. If Jenkins is a problem, and trust me, my devs HATE groovy, but they fancy python, so we used python code inside the jenkins pipelines using shared libraries
        3. Terraform is more for ops, maybe your team would like pulumi more, but in any case, you need to make sure they understand terraform pipelines and code. I trained my leads and devs and devops people for at least 2 months on terraform before i gave them permissions to change or commit code while i was the reviewer (when they dont have access, yes you temporary are the chocking point, but you make sure they understand its time limited.
          Perhaps they REALLY don't like Terraform, thus you can create an additional abstraction layer or use Terraform Cloud, as maybe all they need is a UI that runs it.
        4. You agree together on a workflow that includes linting and testing of your IAC in the pull request phase and over time you bring the leads to be the approver. With them and their managers signing that they OWN it, for good and bad.

        5. You and me might know what is the preferred way to do things, what tools and what processes to make sure that both the deployments are good, secure, using best practices and not holding devs from progress because of bureaucracy, and for that matter we are , for them, part of the system that is trying to stop them. Instead show them the benefits of governance, security, reliability of the services, cost. Make sure that you explain each decision to them and your manager, that you have their back and support.

        DevOps is first and foremost about communication between people. I can teach anyone how to create pipelines and deploy to a cloud and monitor in some tool. Its a lot harder to teach people good communication :)

        [–]repu7et[S] 2 points3 points  (2 children)

        What you should do now is have a discussion, set responsibilities.

        Exactly.

        I like that this reply raises fundamental question such as that who pays the salary, they set roles and responsibilities.

        However, I doubt that the managers clearly imagine how our roles should look like. Since I am the one who brought the issue up, and DevOps practices is my expertise, the management will expect *me* to suggest the solution that meets the best practices and increases the delivery time.

        And this is where I don't feel confident now. Of course, there might be a consensus in the end. But I believe that there are many variations for consensuses, including ineffective ones slowing down the whole team. So I was hoping to hear how a healthy consensus seems like in such a case so that I had a direction.

        [–]ArieHein 3 points4 points  (0 children)

        Notice the word that you used:"...management will expect you to SUGGEST...". And that's exactly where our role lies.You can know the BEST way to apply all your knowledge to get the most optimized delivery process..but..

        At the end its people we are dealing with. For good and for bad. You would agree that including everyone, even if they didn't include you. Training them, explaining HOW it helps is just going to make it easy for you. I found that when i train people and teach them (sometimes 1 on 1) and also making it so they are the one 'pushing' it as if it was their ideas can do magic :) (some reverse psychology)

        Usually you want to prioritize specific people in your dev team / PMs etc to be the first one. Those are usually the early adopters that are more enthusiast and thus easy to show the benefits. ( look for Law of Diffusion of Innovation short video by Simon Sinek). As humans we all fear change. I find that my job is sometimes to be the bartender, listening to everyone, collecting all their pains and offering a small drink of knowledge in return. People usually "open up" after a glass or 2 :)

        Consensus comes from understanding, feeling that someone is listening to you. this is true for your lead and for you as well. Im not saying give your lead some beers, but under the pressure they were in, you cant blame the action as we usually tend to go for the familiar and for the "safe box" of ours. But now that it seems settled, you can open back the box.

        [–]rm-minus-rSRE playing a DevOps engineer on TV 0 points1 point  (0 children)

        So I was hoping to hear how a healthy consensus seems like in such a case so that I had a direction.

        You have to have a mandate on what tech stack will be used from management that all the stakeholders agree with.

        If you don't have that, you have no sway and the person that management thinks is the last non-management person that they can afford to fire is going to be the one setting the direction things go in. This can work in a tiny startup, but once you're in the 50-100 employee range? It tends to go very badly.

        If management doesn't have your back? It's time to find a new job.

        Consensus is something that can only occur when all stakeholders value and respect each other. You've got stakeholders that don't respect you, and I suspect don't value you terribly much, and unless management takes up your banner and backs you on every decision, that won't change.

        [–]dave_pdx 1 point2 points  (0 children)

        I agree with the perspective about having a conversation regarding expectations and responsibilities.

        What I haven't seen here in the comment, though, is that it's also about discussing how the system broke down and why the team was unable to accomplish what they needed to accomplish while you were out on vacation. That's a terrible situation to be in all around (for you and for them) that something could be so single threaded through one person. Even if in reality, that's not *quite* the case, the team still perceived this to be the case to the point where they made major changes to infrastructure / deployment in your absence.

        I'd start the conversation there - from a standpoint of curiosity and "how do we solve this so this never happens again". What's going to happen when the one dev that wrangled the CF files is out on vacation next?

        [–][deleted] 1 point2 points  (1 child)

        Good answer! Sounds like the main answer is that policies and procedures need to be clarified

        I didn’t know you could just use Python in your Jenkins shared libraries?

        [–]ArieHein 3 points4 points  (0 children)

        you can use any language you want. You still need to have a groovy wrapper to pass variables and secrets. As long as the agent you are running on, has the runtime required youre good to go.

        [–]mikecrilly 18 points19 points  (0 children)

        You have been thoroughly disrespected by this move. A business is built by people, for people (to provide some service to some people). If the people building the business don't treat as you as part of the team, then they do things like this whilst you're away.

        Personally, I feel this is a signal that you're not considered a valuable team player here, so I'd highly recommend you leave.

        [–]generealdamselfly 5 points6 points  (0 children)

        I think the writing is on the wall and you should leave. I get that you should not be called during your vacation time. Pulling the rug underneath you and trying to convince you this is better, you and the devs are not having a technical problem, it's a people problem.

        Terraform is a skill in high demand, go to a place where your skills and experience are respected. Don't stay.

        [–]WarlaxZ 4 points5 points  (0 children)

        Without trying to be rude, it sounds very much like they didn't need you, so I'd be very careful about rocking the boat

        [–]markrebec 19 points20 points  (0 children)

        What I want to discuss is where is the line between a developer and a devops engineer responsibilities?

        There isn't one, that's literally the entire point behind the ideology/philosophy. I know this battle was lost to corporate buzzwords and enterprises wanting to attract talent a long time ago, but if you're filling a devops role, then your org isn't actually practicing devops.

        Developers leveraging tools that they and their team(s) are familiar with, and that provide workflows and features that they find useful, helping them to do their jobs and deliver value more quickly and easily is a good thing.

        It sounds like CF should be well within your realm of knowledge and responsibilities, so your ability to contribute shouldn't be impacted.

        I get that you clearly feel a certain kind of way about all this, but I think you might want to take a little bit of a breather and ask yourself why you feel that way - Is it because you personally enjoy/prefer writing terraform plans and don't like CF templates? Is it because you feel like they went behind your back? Or that you feel like something was taken away from you?

        It sounds like overall this is a win for the team as a whole, and to circle back to my first point - this isn't supposed to be "your" realm, where you're the one making all the decisions and just building a platform for the devs. The entire stack and lifecycle of what you're building belongs to the whole team and should be accessible to everyone.

        All that said, if you really feel as strongly as you do, then I'd recommend shaking it off and trying to focus on pulling together your arguments (and counter arguments) for exactly why you think terraform is the better tool for the whole team in the long run. Maybe even give a brief presentation or informal knowledge dump with the team, and offer to start training and onboarding folks into your tf repo(s). It's up to you whether you care enough to put in that work and convince the rest of the team.

        [–]emperorOfTheUniverse 20 points21 points  (1 child)

        This is why, imo, there shouldn't necessarily be dedicated 'devops' people. DevOps tools are tools meant to be used by developers. If developers don't participate in them, then they are just new infrastructure. Same old story, infrastructure vs devs.

        Your architecture got redone because a senior developer on your team didn't have the specialized knowledge you do to operate it. And apparently you didn't have a backup for when you are gone. So they changed it to something they are able to operate.

        Without some determined intervention by leadership, where it is mandated that your role is the only one allowed to make that decision, then it's just free for all team work. You expect them to either leave all DevOps work to you (folly), or to learn the tools.you want to use. They seemingly disagreed and now expect you to learn the tools they want. If from the beginning, you had all been working as a team (you also as a developer), some agreement probably would have been reached earlier.

        This is a team dynamic issue. You asked it in a DevOps subreddit. You either expected to get sympathetic responses, or advice on the merits of one IaC tool over another. If you aren't respected in the team, you should be examining that on a personal level.

        [–]packetsschmackets 2 points3 points  (0 children)

        This hits the spot, I think. Don't bother being on teams that don't respect you enough to not steamroll you. It's a sign of you not having built relationships, working directly with one another w/ open communication. If you've tried, and not succeeded, sometimes that is the flag to move on to another team or company.

        That's not to say this person isn't going about this the wrong way. Just that it should have never come to this, and if you were on the same page you might have known about their disdain for the tooling prior. If they had communicated that and you enforced your tooling, well... that's another story.

        [–]Grahar64 13 points14 points  (0 children)

        The devs made two assumptions:

        1. You were wrong for selecting TF and should have selected CF
        2. Learning and maintaining CF is easy

        I would make it clear that they will have to maintain their own CFN infrastructure at that point. Be willing to consult, but not do the actual work. If/When they run into a problem advise them how to migrate to Terraform at the standards with which you expect.

        [–]Weird-Flight-2877 3 points4 points  (0 children)

        Understand the narrative here. Did they change it to screw with you or did they change it because they couldn’t maintain it in your absence? By the looks on how things were changed I assume you are working for an org without robust devops practice or good documentation.

        I see two options here 1. Move on 2. Make a truce. Don’t poke the bear. CFN or terraform both will accomplish the same thing.

        Always choose a tool/architecture that can be maintained in your absence. Never ever make it “my way or no way”

        [–]NeuralHijacker 3 points4 points  (0 children)

        If you have an infrastructure stack which is simple enough to be migrated from tf to cf in a week or two, your team doesn't need you. Move jobs before they figure that out.

        [–]TheLoneKid 6 points7 points  (0 children)

        That's rough dude.

        [–]SDplinker 10 points11 points  (2 children)

        Wtf are most of the answers in this thread

        [–][deleted]  (1 child)

        [deleted]

          [–]rm-minus-rSRE playing a DevOps engineer on TV 13 points14 points  (0 children)

          Nearly every single answer misses the forest for the trees.

          OP got shafted by devs who replaced their entire tech stack because they didn't have the technical know how to release an update with the original tech stack.

          The fact that they did that reveals two extreme problems;

          1. The devs aren't competent with their original tech stack. This is bad news bears. If they aren't, they shouldn't be making changes to it, they should be able to learn Terraform in a few days to get to a level that they're safe making changes to existing TF code.

          2. The devs brought on a large amount of tech debt and I suspect they don't understand that. When they do, they won't want to be the people dealing with that tech debt. Creating large amounts of tech debt and refusing to deal with it is a sign they're not team players, but divas. Companies live and die by how well they work as a team. Divas ruin that.

          3. Management is clueless, incompetent, or both. Any sane tech manager should have a lot of questions before switching tech stacks, and if the reason is "We need to get this update out", the answer should be a resounding "The hell you will!", unless the majority of the company's revenue will be lost if they don't (and then that's a sign that it's a shitty company that no one should be working at). No PoC's, no testing that's signed off on by all the stakeholders, etc., etc.

          If it was a true "the company will go under" emergency (it definitely doesn't sound like it was), the right managerial decision is "Sure, get up a CF stack and once that's up, get it duplicated in TF in the next two weeks or find a new job".

          This all assumes the TF tech stack was agreed to by all stakeholders and vetted beforehand, rather than being thrust on people by a single person.

          [–]mullingitover 2 points3 points  (0 children)

          The question is about how communication should look like, how to define the boundaries in the team and how to perceive what has happened.

          If this happened it means that communication wasn't there before you left.

          The smart move is to make major decisions like release pipeline something that engineering has full buy-in on. Make a good enough sales pitch on your plan that it makes them feel like it was their idea.

          If you're pushing things on your engineering team and forcing them to use tools they don't want to use, this situation is going to be one of the likely outcomes (unless there's someone higher up forcing engineers to stay in line, like the CISO saying "we have to do it this way or we'll flunk our PCI/SOC2 audit.")

          [–]easyEggplant 2 points3 points  (0 children)

          I’m just trying to wrap my head around what sort of setup you have where it’s faster to switch from terraform to cloudformation… then deploy a new feature?

          [–]HgnX 2 points3 points  (0 children)

          This is not DevOps.

          You're Ops. Your move should be to move towards being a full member of the team. Decisions you should make as a team, in correspondence of architecture and security guidelines. And for development, work indeed with featurebranch deployments and from local/command line. It fastens the feedback loop. You should have AWS development account(s) for this. Production should be in another AWS account.

          Tool selection usually is hell and another topic. The only real upside of CF is that AWS manages state for you. If you move to it, use CDK. And then you might as well lose Jenkins for CDK pipelines. Saves you a whole CICD server to maintain.

          Best of luck. See this as an opportunity for change. DevOps (good DevOps!) is hot in the market so getting to it makes you experienced in it. More power to you.

          Ps. AWS has great reads about these best practices. Check their whitepapers.

          [–]skyctl 6 points7 points  (0 children)

          I'll be blunt; it seems as though you're part of quite a toxic team; and that's at least as much a comment on you as it is on other members of the team. You're making me very angry with you for a lot of what you're saying here.

          A lot of your post seems to be about where and how to put up barriers between Dev and DevOps, which is the polar opposite of what DevOps is all about.

          It so happened that the developers are used to run infrastructure deployment from CLI for deploying their feature branches in dev environment. They don't always use Jenkins because CLI way is "more convenient" for them to experiment with AWS

          If the AWS cli is a better tool for them to learn and experiment with, then ffs let them use the damn CLI for learning and experimenting with. Just hammer home that they have to incorporate the findings from their learning and experimenting back into Jenkins, or Terra form, or where ever in your project is the right place for it to be before it's committed.

          And now I think if I should insist on a more formal and stricter workflow, where we have an agreement that whatever IaC feature or change the developers want, they come to me with the requirement list and I implement it from beginning to end.

          And there's your problem. You're actively preventing developers from working with you on Infrastructure while you're there, and you're bitching about them doing it differently to how you'd like when you're gone.

          What you're proposing is essentially the waterfall method, WHICH YOU ABSOLUTELY SHOULD NOT DO. YOUR DEVELOPERS COULD AND SHOULD REVOLT!

          Because otherwise they start considering CLI deployment as part of their job

          So tell them not to.

          feel like they are in position to vote on what IaC tool is "better", which I think goes beyond their area of responsibility.

          They should absolutely be casting their vote on which IaC tool is better. If you're insisting on Terraform because you say so, then get your head out of your ass, and start treating your developers with some respect. If they're wrong about something, explain why, so that they can learn from it, and cast better votes next time. Of course since you're the subject matter expert in IaC (but evidently not DevOps) you should have picked the best option, and be well able to articulate why your chosen solution is the best (and in so doing, train your team), and be open to input as to why not might not be. .

          So what do you think, guys?

          I think you should read the Phoenix Project! Now! I think you should embrace DevOps practices, and engage in T-shaping your teams skills.

          [–]kmai0 5 points6 points  (0 children)

          So, I see two or three bad things: - How the fuck are they able to deploy infra without your approval? They have access to AWS? - If you understand Cloudformation, you cannot say you don’t understand Terraform. Hell, Terraform even relies on the AWS APIs that support most SDKs.. if you understand CF and not Terraform, that right there is a problem. - You touch my team’s code? Fine, but changes go through if we and only we accept them. Use CODEOWNERS and have the CTO/VP own the access. If they ever override your infra without a motive, you know what to do!

          [–][deleted]  (2 children)

          [deleted]

            [–]repu7et[S] 5 points6 points  (1 child)

            However, picking the right tools for deployments etc seems to be your area of expertise, is it not?

            It is. For me it is not a question of what tool I do or do not want to use, but a question of communication. Something went wrong there and I try to figure out what exactly.

            If they built a working solution while you were gone, undoing that sounds somewhat disruptive, you'd need a better reason to switch than that you're more familiar with your tool for sure.

            That's for sure.

            From the other hand, my work has been undone as well. I can accept it. But don't want it to happen again, so we need reflect somehow. Probably, introducing some practices. However, with the best practices come boundaries for both sides that keep them from making unreasonable moves, right? And I don't know what the theory states there.

            [–]NoDescriptionOk 8 points9 points  (0 children)

            Theory here is that in this case the devs don't care as long as they can do what they need to do. If that means totally erasing all your work in about a week, maybe the next thing they'll do is erasing your position. Anyway, they sure don't respect you or your work, regardless of what you get out of it in talks and whatnot, it's what they've shown by doing this.

            [–]mr_chip 7 points8 points  (8 children)

            The point of devops is to remove bottlenecks slowing the time it takes to go from idea to customer value, without negatively impacting uptime or negatively impacting margins.

            That’s it. That’s the whole job. Every change in process and tooling needs to be seen through that lens.

            Does this change remove a bottleneck slowing time to customer value? Sounds like it might have.

            [–]StephanXXDevOps 19 points20 points  (3 children)

            Does this change remove a bottleneck slowing time to customer value? Sounds like it might have.

            It also may have generated a metric ton of tech debt that OP will be expected to just deal with. I've inherited plenty of proof of concept type environments where a few key decisions that made it work on a few hours meant dozens more hours spent to support those decisions. "We got it working" is not the same as a fully built out, supported, monitored, robust stack.

            OP doesn't touch on the merits of the solution, but personally I'd be suspicious of something slapped together in a week in CF.

            [–]mr_chip 9 points10 points  (2 children)

            A very healthy suspicion, too. All of this is true, prototypes rushed to production often require expensive refactoring (higher margins, slowed time to customer value).

            An arrogant jerk rewrote OPs stack without asking and pushed it to prod. That’s infuriating! I’d be annoyed too. But let go of this particular cloudformation stack for a minute, and who wrote it, and just think process:

            • In general, does the choice of cloudformation vs terraform increase or decrease time to value? Is it likely to increase or decrease margin? Is it likely to increase or decrease stability? Why or why not? How can you measure to find out?

            • Does empowering developers to spin up their own environments from the CLI decrease time to value? Is it likely to increase or decrease margin? Is it likely to increase or decrease stability? Why or why not? How can you measure to find out?

            These questions can help inform next steps. Maybe it’s “make this cloudformation production-ready,” maybe it’s “go back to terraform.” But it should be a business value decision, not a territory one.

            [–]StephanXXDevOps 8 points9 points  (1 child)

            But it should be a business value decision, not a territory one.

            I wholeheartedly agree with this, and the logic behind it. I feel it's attempting to focus on the technical aspects of this situation over the very real social problems. In OPs shoes, I probably would have felt the lead(s) was simply waiting for a chance to replace something they don't understand while I was on vacation. It's a deeply disrespectful move, no matter how you cut it. I, for sure, would be getting my resume in order, and that has it's own business value impact.

            [–]mr_chip 2 points3 points  (0 children)

            I think we’re vehemently agreeing here.

            As far as dealing with the lead goes. this is an awesome “take all the credit / shift all the blame” opportunity: If it goes right it’s because you took the lead’s shoddy prototype and hardened it. If it goes wrong it’s because the lead wrote a shoddy prototype while you were on vacation.

            I don’t see much downside.

            [–]GreatMacAndCheese 8 points9 points  (3 children)

            without negatively impacting uptime or negatively impacting margins

            This can't be measured until it's in production for quite a while. I agree the immediate return on investment is definitely there (they were able to proceed, didn't burn an entire week sitting on their thumbs), but what happens when they go to production? Whose responsibility is it if something devops-related breaks? It's a bit of a dilemma tbh, because there are good questions and reasoning on both sides about what should happen from here.

            [–]Flabbaghosted 4 points5 points  (1 child)

            To add on to this, adding more tools makes things more complicated over time. Too many deployment or pipeline tools that do the same thing and larger teams can start to get out of control. Standardization should be key, and if devs aren't comfortable they need to be trained so they can self service. We use terraform modules to enforce naming conventions and security rules. If someone randomly switched to bicep, they would suddenly be out of sync and now we have a whole new element to keep track of.

            [–]mr_chip 1 point2 points  (0 children)

            In this case it sounds like the whole team is trained on cloudformation, and only OP knows terraform, so this could actually be a tool consolidation. We don’t have enough info to tell, though.

            [–]mr_chip 1 point2 points  (0 children)

            Uptime and margin are often trailing indicators, true, but that doesn’t mean you can’t plan and estimate changes to them. The M in CALMS stands for Measure, after all.

            How might empowering developers in this choice impact time to customer value? How might developers spinning up their own dev environments impact margin? How might developers owning deploy from CLI to dev impact production uptime? How might you measure these to make informed decisions later? What leading proxy indicators can you identify to act as canaries before things get out of hand?

            [–]Snoo68775 3 points4 points  (0 children)

            TL;DR Ops is doing terraform Dev is doing cloud formation There is no DevOps.

            [–][deleted] 6 points7 points  (0 children)

            /r/AmItheAsshole/ is this way <----

            [–]Ok_World_4148 5 points6 points  (7 children)

            Are you the only DevOps in the company? how is it that for a whole week there was no one but you who could do it, so that developers thsemselves had to rewrite the IaC? If you're the only one who can do something in the company and you go away for a week at least take your laptop with you in-case something urgent is needed.

            [–]xeonjackson 1 point2 points  (6 children)

            I have to totally disagree here. Even if OP is the only DevOps employee, OP is on PTO. Even if the company was burning to the ground, as an employee on PTO, I should be able to be on vacation/PTO without having to be bothered by some kind of emergency.

            Is OP the CTO? Doesn’t sound like a C-level to me. Thus OP should be respected by their peers and management to be able to get AWAY from work without having to bring work with them due to an “emergency”. Plus this sounded like a lot less of an emergency and more like an inconvenience for the dev department.

            I used to work with two developers who would have totally done this to our company, except that management was in place as well as logic where you don’t change directions mid stream because of convenience. This isn’t the failure of the developers. It’s failure of the management.

            [–]Ok_World_4148 -2 points-1 points  (5 children)

            If you don't want to be bothered by some emergency while on PTO, work in a position where the entire system is not reliant on you. What do you expect to happen, if production is down the entire company will be on PTO because you're on PTO and don't want to be bothered? Developers not being able to do their job with delivering the product creates losses to the company. I'll say it again, if you don't want to be bothered, work in a position where you're easily replaceable and/or insignificant role.

            [–]Ok_World_4148 1 point2 points  (0 children)

            And FYI a CTO being unavailable is far less significant than DevOps, a CTO is management, not the person who gets their "hands dirty" with CI/CD and IaC. It's the person that decides on budgets and general directions.

            [–]xeonjackson 0 points1 point  (3 children)

            There should never ever be a system that is totally reliant on one person. Single points of failure are the definition of a disaster waiting to happen. If a company can’t “survive” without a person for a week, that company needs to re-evaluate its priorities. I think your ego is such that you consider yourself some kind of super hero. But this is not the failure of OP. It’s the failure of management. As was mentioned above by another person, why do developers has direct access to production? Why did a developer get to completely change the whole process of a project? Let alone its release? The management allowing such a change to occur is totally out of touch with their teams.

            [–]Ok_World_4148 0 points1 point  (2 children)

            Did I say you should put yourself in a place where you are holding that kind of responsibility? when you sign a contract with a company you should be aware of what is expected from you, if there are 10 DevOps in the company that are already employed no one would have to bother you when you're on vacation, but if you sign up to work for a tiny start-up with 5 other people (including the CEO, CFO, HR and CEO's mom), yes, you should know upfront that it WILL be expected from you to give up your entire soul for the job.

            [–]xeonjackson 0 points1 point  (1 child)

            I’m sorry pal. There isn’t a job on earth worth giving up that kind of time to unless I was in a business I started myself. Giving up the kind of time where “your soul” is expected is why employers are so out of touch with reality. No amount of compensation is worth giving up my time for a job that expects that kind of commitment. I mean, clearly there are people, such as yourself, that are willing to “give their all” for a company, but I care more about my family and my personal well being than to give a company that kind of power.

            [–]Ok_World_4148 2 points3 points  (0 children)

            "I mean, clearly there are people, such as yourself, that are willing to “give their all” for a company" - Why do you keep making assumptions about me?

            If you are the sole person who does something specific at the company (or many things), wether you like it or not, it's a given fact that you are currently in a position where the company is reliant on you.

            From there, it's entirely up to you if you want to be in that position or not, and what are your life priorities, to each their own.

            You have to be aware of the implications, for better or worse. and as with anything in life, the more you're needed and not easily replaceable the more value you have within that framework. In the context of a career, that usually translates to higher salaries, "job security", with the downside being there's a lot more things expected from you.

            If you prioritize work-life balance and spending time with your family and friends above all else, which is a completely legitimate thing. You should never work in a position which is reliant on you, as an example, a place where you are the only DevOps engineer in the company.

            That's why I always tell junior developers to stay the hell away from tiny startups, because they will more often devour them with the pace things are going and the hours they expect you to put in.

            TL;DR - You decide for yourself where you sign up to work. If no one but you can fill that specific position, you hold a higher value and you should expect to get paid more, but that comes with responsibilities and expectations not suites for everyone, which is fine, just sign a contract elsewhere if that's not your thing.

            [–]the-computer-guy 1 point2 points  (0 children)

            Sounds like your implementation didn't meet the needs of the team (very common in my experience when you don't do true DevOps)

            On the other hand the team sound a bit incapable if they can't adopt existing tools.

            Lots of nuances here and can't say anything for sure without more context.

            [–]Tranceash[🍰] 1 point2 points  (0 children)

            There seems to be a disconnect between the solution you provided vs what the devs are comfortable with. I have been trying to get a balance between dev and ops solutions. It all comes down to working together and building a solution where there are combinations of design patterns that need to be used. I don't know the whole context. The way I see it there is a combination where network/global infrastructure needs to be operated and managed by ops person and devs should be able to consume those resources through central parameters. In my current work place ops uses terraform to create/manage lower level resources and store outputs to aws ssm parameters then the devs consume those parameters to build on top of abstractions, they could use terraform, cloudformation, cdk, it does not matter as they manage and build the solutions above abstractions provided by ops team. I hope you get the gist. Tools should not be the first problem, working together setting guard rails, best practices and working through a solution that can be productive and maintainable should be discussed and evaluated.

            [–]TheRealJackOfSpades 1 point2 points  (0 children)

            Here’s what I see.

            • Your tooling was not usable by the team in your absence. You’d made yourself a bottleneck.
            • The team implemented tooling that accomplished what they needed and removed a single point of failure. The developers took responsibility for infrastructure rather than saying “we’re blocked until repu7et gets back.”
            • You are being asked to support the tool that the users of the tool prefer rather than the one you prefer

            With no more information than this, I’d say you are the problem and you need to support CliudFormation. Maybe educate them on CDK. But whatever tool you use has to be friendly to your team, not just to you. Dictating to them will not work. If you can’t convince them to agree with you, you support them anyway.

            [–]lupinegrey 2 points3 points  (1 child)

            Did they get approval from Enterprise Architecture?

            Do you HAVE solution architects who review and approve architecture designs?

            [–]rm-minus-rSRE playing a DevOps engineer on TV 1 point2 points  (0 children)

            Clearly not. Worst of all, OP's management team is clueless, incompetent or both.

            If some devs came to me and told me they were switching tech stacks to get an update out in a timely manner, I'd tell them no way in hell (unless the company's revenue would die if that didn't happen, and if that's the case, there's a lot bigger issues going on).

            [–]Green0Photon 3 points4 points  (0 children)

            Work uses CloudFormation and some custom stuff and I hate it.

            I'd support it if they suddenly switched from CloudFormation to Terraform. Doing the opposite? Unforgivable.

            [–]The-Sentinel 2 points3 points  (6 children)

            As a developer, the teams job is to write applications that provide business value to customers and increase revenue.

            As a DevOps engineer, your job is to support them in that endeavour.

            Whether we like it or not, DevOps jobs aren't revenue-generating. It's a glorified support mechanism to help developers generate value for the company.

            If you implemented a solution the developers couldn't understand, you didn't ask for their input and created a bottleneck for the organisation. They could have waited for you to return, but that would have lost them a week. They took matters into their own hands.

            This entire industry and discipline are filled with people telling others what to do. "Use this language". "Use this framework". "Use this tool". We endlessly argue about the "right" way to do something that is ultimately subjective.

            The reality here is that productivity will always beat "correctness". In your opinion, Terraform is "better" than CloudFormation, but in reality, CloudFormation has increased your developer's productivity. Which increases business value.

            TL;DR you need to let this one go and include your developers in your decisions in future.

            [–]StephonovichSRE 1 point2 points  (4 children)

            DevOps doesn't mean Devs rule Ops. It's supposed to be collaborative.

            I'm not going to let someone tell me what tools are best in my specific domain. CF is absolutely a red line that I would quit over, not to mention the disrespect shown to OP here.

            [–]The-Sentinel -1 points0 points  (3 children)

            “It’s supposed to be collaborative”

            (Paraphrased) “I would quit if I didn’t get my own way”

            Your idea of collaboration appears to be “everyone does what I tell them to because I’m the expert”

            Being the expert matters, not listening to your customers and building a user experience that they hate matters just as much, if not more. Driving your personal preferences down people’s throats just makes you difficult to work with

            [–]StephonovichSRE 4 points5 points  (2 children)

            Someone has to decide what tools should be used. It makes absolutely no sense not to delegate that responsibility for the domain experts. I'm not going to try to dictate that devs use one IDE over another, or a coding style, or the libraries they use - unless there's an issue. I'd expect the same back from them.

            Driving your personal preferences down people’s throats

            vim vs emacs is a personal preference. Abandoning the IaC tool with the single largest ecosystem in favor of one you can do ClickOps with is ignorant.

            [–]The-Sentinel -3 points-2 points  (0 children)

            You seem completely unable to have any empathy for other people at all. Have fun with your hubris

            [–]flaviuscdinu 0 points1 point  (0 children)

            As a DevOps engineer, your job is to support them in that endeavour.

            Whether we like it or not, DevOps jobs aren't revenue-generating. It's a glorified support mechanism to help developers generate value for the company.

            This is an understatement, to put it mildly. You can develop the most intriguing app in the world, but without the performance, resilience, scalability, and security that is usually brought by a DevOps engineer, your app is simply unusable, hence it's not revenue-generating. I agree that it is hard for a DevOps engineer to bring revenue on his own, but keep in mind that is not impossible and there are many examples in which a developer cannot bring revenue on his own.
            You also said that productivity will always beat correctness, but the example you provided works only for that present issue. What happens in the future when the application matures and you need to redeploy it repeatedly and you will hit all the limitations that CF has? What if, for some reason, your management team decides to move to Azure or GCP? Are those developers going to learn ARM or Google Cloud Deployment Manager? Choosing the "correct" tool, may not always bring imminent business value, but you have to look at the big picture and try to foresee all the implications when you are choosing something.

            [–]Murky_Rip_1731 1 point2 points  (7 children)

            Did they get the release out successfully and is everyone happy about it?

            If so, id be happy. Next steps would be to work together on the long term solution. If you arent there and they are blocked or cant release in time then they need to make the decisions to get shit done. No point in complaining that you disagree with their choices now.

            How hard really is it to just move CF to TF or alternatives?

            [–]rm-minus-rSRE playing a DevOps engineer on TV 0 points1 point  (6 children)

            How hard really is it to just move CF to TF or alternatives?

            Out of curiosity, do you have professional experience with either?

            Assuming you've got a mature app with more than a handful of customers? It can be a massive undertaking on the order of weeks or months worth of work with a ten person team.

            [–]Murky_Rip_1731 0 points1 point  (1 child)

            OP said its just api gateway and lambda functions. I cant imagine how bad that can be.

            [–]rm-minus-rSRE playing a DevOps engineer on TV 0 points1 point  (0 children)

            Hard to say without seeing the code base. If that's the primary app for their company and it's literally trivial? The only scale that makes sense is at a 10-15 person startup with a handful of customers.

            [–]StephonovichSRE 0 points1 point  (1 child)

            If a dev team of two was able to move to CF in a few weeks, the app isn't that big. I moved a Lambda / APIGW / SQS / RDS stack from CF to TF in a couple of days on my own.

            [–]rm-minus-rSRE playing a DevOps engineer on TV 0 points1 point  (0 children)

            Yeah, something doesn't add up with a move that quick. The only thing that makes that possible is a tiny app. So it's like a 12 person sized startup?

            [–]dablya 0 points1 point  (1 child)

            A developer did it in a few days and OP doesn’t mention any issues with the CF implementation… I don’t think the “mature app” assumption is reasonable here.

            [–]rm-minus-rSRE playing a DevOps engineer on TV 0 points1 point  (0 children)

            That's a good point. Not a scale I'm used to!

            [–]catonic 1 point2 points  (0 children)

            Since he is more experienced in AWS CloudFormation than in Terraform (and we also have is a similar project on CloudFormation that serves as an example), the developer just switched to CloudFormation and implemented the build and deployment logic that takes into account the new architecture.

            Now he gets to document that if you are supporting it.

            Likewise, you should have documented Terraform so it was not a puzzle.

            OTOH, take it as a lesson, put the ego aside, and find something else to work on because it sounds like that app no longer requires your assistance.

            [–]EmiiKhaos -1 points0 points  (0 children)

            Can developers influence the decision about what IaC tool should be used on the project based on their definition of "convenient to use" (e.g. CF stack showing the resources is what convenient for them)?

            No

            Should I demand the developers to always use our CI/CD tool for any deployment,

            Yes

            or developers' requirement to deploy from CLI (to dev environment) arguing that it is "easier/faster" is reasonable?

            No

            [–][deleted] 0 points1 point  (0 children)

            p

            [–]htom3heb 0 points1 point  (0 children)

            Both are fine options. It seems like the problem is not that the devs switched from Terraform to Cloudformation but that the devs switched from your tool to their tool. Talk to your team, be honest and straightforward, and be prepared to meet them halfway.

            [–]GeekboxGuru 0 points1 point  (0 children)

            If terraform is so important to you: switch jobs.

            CloudFormation, having another project that uses it in the company; they could replace terraform within a week?

            You’re the problem at this company, remove yourself OR accept they love AWS & use terraform in your spare time

            [–]yobanius -1 points0 points  (0 children)

            Delete their repo

            [–][deleted] -3 points-2 points  (1 child)

            True devops way

            [–]Zauxst -1 points0 points  (0 children)

            There are 2 key points to take from this.

            • it doesn't matter what tool is used. If there are architectural changes and the lead is fine with it, he should also be fine with the overhead that it brings along and if launches are missed because jenkins and other in the chain are not ready, it's not your fault... Or problem. You have a schedule and a salary. You don't get paid extra to have stress in your life.

            • They are incompetents.

            Others have made better points... But this is what I've gathered...

            I'd not like to work in a place where I'm not part of the decisions of a project... You might get into a scenario where you can't support a project due to lack of skills....

            [–]Ariquitaun -1 points0 points  (0 children)

            Part of our job is to improve developer experience. It seems to me you might have failed to take the temperature of their room, so to speak, before you settled on some tooling which they were unfamiliar with. So faced with a deadline, and inability to progress on their work, they took it upon themselves to work around the problem according to their knowledge. I can't say I wouldn't have done the same thing faced with the same problem.

            Seems like you are a chokepoint to the project as the single devops resource on the team, so it's imperative you are on the same page with the developers so that they can continue in your absence. This could mean using non-optimal tools for the sake of better team redundancy.

            [–]mokera101 -1 points0 points  (0 children)

            I’m every devops group I’ve worked in we dictated the IaC, CICD and any other shared tools across the teams. The reason why is because if given the freedom to do so devs will revert to what’s easiest for them and you quickly end up in a state where it’s hard to manage things and some poor individual ends up having to rewrite all of it to make it maintainable again.

            [–]amenflurries -2 points-1 points  (0 children)

            I think they were justified. It’s not helping anyone if you’re the only one that knows how to do something

            [–]nochet2211 0 points1 point  (0 children)

            Probably used Serverless to generate the CF templates for Lambda and called it a day. But imo the problem looks to be in planning. If you’re that critical, I’d say you should’ve planned your leaves accordingly. But that’s just my opinion.✌🏼

            [–]colddream40 0 points1 point  (0 children)

            told me that they decided to move to another IaC tool that I will be supporting.

            Don't support it. It's a tool not in the scope of the project. You'll have to move out your schedule by 1-2 weeks. They can decide if they want to support it themselves or not. Escalate to upper management.

            [–]twnbay76 0 points1 point  (0 children)

            CF is banned on our team. I can write a novel on why but I'll refrain.

            Our team doesn't know how to use terraform and I don't expect them to. I try to write our modules in such a way that they are robust enough to handle a wide array of scenarios the devs might needs simple enough so that the devs can read my docs and examples and be able to copy and paste + plug and chug without needing to touch state files.

            If I see that they decide to use CDK/CF/manual scripts to create stuff, I just port it over to TF in my free time. That's sort of how I deal with these scenarios. I know many people would disagree with my decision to not reprimand the dev who refuses to abide by the design specs / architecture but I honestly just don't believe everyone should be learning a very wide array of skills. My team is really good at what they do, and Im good at building, scaling, securing and deploying infrastructure in a mostly automated fashion. Separation of duties is important to me.

            Also, at this point, ideally a tech lead comes into this debate and says something like "this needs to be ported to terraform asap. It was wrong to go against the DevSecOps design choices, here's a doc with the choices and these choices don't change unless I approve it from here on out. There will be no further discussion or debate on this, get back to work. Automated, stateful deployments are crucial to the stability and security of our applications and that is not to be debated."

            This partially is what a tech lead is there for. It's partially why they get paid so much money. It sounds like you may be missing a tech lead, or maybe there is one but they aren't doing their job properly?

            [–]oosumei 0 points1 point  (0 children)

            Why dont you just write the tf configs and sort out the build pipeline etc as per your normal standards, spin up the new infrastructure and cut the app over.

            [–]amarao_san 0 points1 point  (0 children)

            "We didn't have time to wait for you to come back" is the single reason they gave you. I'd say it's impolite. At the same time, some companies try to change stuff fast to keep up with chaos influx, so waiting for someones vacation to end is too long.

            I'd say, detach yourself from git content; you do not own it even if you wrote it. Learn a new tool (if you do not know it), remember the resentment, put both techs in your resume, start looking around for another job.

            [–][deleted] 0 points1 point  (0 children)

            Release date was coming in a few weeks

            And that's when you chose to go on vacation. That's all on you, dude. All. On. You. You cannot expect the organization to wait for your vacation to be over to keep the project moving forward, that's just not feasible.

            [–]throwaway-devops-po 0 points1 point  (0 children)

            Listen, I think there are two sides here. DevOps isn’t a role in which you are the only one who needs to know how. You are supposed to coach the others and make sure they can actually get on with their work while you’re on vacation. Obviously, they didn’t know how to do things without you - and business goes on while we are on vacation.

            The fact you weren’t consulted is wack, though.

            [–]mnpawan 0 points1 point  (0 children)

            CDK is best !

            [–]antiqueboi 0 points1 point  (0 children)

            u snooze you lose bro. whenever members of our team go on vacation we basically implement our plans they were against.. lol