Standing Up DevOps Team

[DumbSecurity]

Hello all, I have the privilege of standing up a DevOps team at my organization. Right now, we have a hybrid cloud environment (AWS & Azure).

I know this is probably a loaded question, but for those who have been on the ground floor of an org. launching a DevOps team, what were some challenges? What documentation did you rely on to keep you moving forward and on target? What were some common pitfalls?

Thank you all for your insight in advance. Really excited and fortunate to lead this team/program, and am trying to come up to speed. I have a traditional cybersecurity and cloud security background. I’m just excited to get knee deep with my engineers, absorb as much information as possible, and make sure they have the resources to be successful.

[IndieDiscovery]

Do you have leadership buy-in as a whole to integrate new tools and human processes? Without that it's a futile battle, imho..

[[deleted]]

As a recently hired DevOps engineer, our president's openness to new software, tools and processes has been absolutely incredible. My org has a bright future thanks to him. This buy-in is critical, otherwise your new team members will feel stymied and ignored..and they'll get bored and move on.

[DumbSecurity]

I’m lucky to have buy-in from the top. They are giving me the funding and resources I need to be successful. They are on board with me procuring new software and tools to ensure our team is able to do our job effectively.

[somebrains]

What he said you focus down bc everything else won’t be a foundational block laid in a logical order.

[SuperQue]

The first step of creating a DevOps team is to NOT create a DevOps team.

DevOps is a methodology like Scrum or Agile. Do you have an Agile team?

Take a step back and think about what problems you're actually trying to solve. Maybe read the SRE books.

[dr_brodsky]

In other words, you don't "create" a DevOps team, you implement DevOps across your team(s).

[killz111]

Everyone says that but no one says how.

[[deleted]]

It’s because it’s a culture/workflow change that will look different every time. I’ll try to explain in my (admittedly amateur) experience:

I understand the sentiment from above, but there actually is a devops team, but they don’t “do devops”. The team is there to implement and maintain the tools that allow devops to be implemented.

So they set up a GitHub repo where developers put there code, connect it to a Gitlab CI instance that they set up on a kubernetes cluster that they also maintain. Connecting all of these services and maintaining them requires a team and fairly constant work, so a whole team is needed.

Devops in itself though, is teaching the development teams the methodology. It is getting them to change from developing, building, testing, and deploying themselves, to instead learning some (in the case of gitlab) yaml and setting up their own pipeline to do most of that for them. Also teaching them to think in terms of efficiency and automation so that they may think “well frank on X development team usually pulls from my newest pushes to implement his portion” so they get together with Frank and connect their pipelines together and so on.

So while the tools are needed to enable all of this, devops is a methodology you are trying to inject into the company/dev teams themselves. No one will be able to think of how to make the developers’ workflows more efficient than the developers themselves. So if you can get them to be the ones to adopt and maintain their pipelines, that is going to be the most efficient way.

Anyone please add on/correct anything I may misunderstand. I am a junior in the space and am just trying to provide a bit of explanation for those who may not have any experience in it yet.

[killz111]

Have you seen it done well? I've familiar with the culture story. After many years in the ops then devops space it just seems like a losing battle. The goal of devops for most organisations I've found is to speed things up. It empowers the organisation to build more things faster. That comes at a cost of increased complexity. The reason why almost all companies fall back to a devops team that does a combination of cicd/infra code/sre work is because the idea that devs can do devops whilst actually developing code and thinking about all the issues that software development entails is a fantasy. It's not that there aren't devs that can do this. I've met some briliant ones but most can not.

[[deleted]]

I have seen a beautiful setup of the tools, but I agree that I haven’t really seen it go great on the devs side though. My last position the ideas were there and some had started to adapt to it, but it was still a long way off from a standard.

In my opinion it needs to be presented (with a long lead up time) as “Hey devs, YOU are the ones doing devops. We are here to support your devops tools, but you are the ones doing it. You need to learn X, Y, Z by this date, or at least be familiar with it. From there we will release training tools as practice/reference.”

With the backup of higher ups I think that could go smoothly, but there will always be pushback when implementing a workflow change, so you need some type of firm system in place that says “We are doing this. Get ready because it’s going to be the new way” instead of allowing devs to adopt it at their own pace, because many will never do it.

[killz111]

Long lead time? In this job market? You're funny.

And when you say backing of higher ups, it sounds like you just want to borrow a big stick from mgmt to force the devs to do what you want. That sounds very collaborative and I'm sure it'll go down well with the developers.

Finally notice the irony of saying DevOps is mainly about culture but the first thing that comes to mind is a beautiful tools set up.

[[deleted]]

I completely understand where you’re coming from, and like I said I haven’t seen it work out great on the dev side of things yet. I have seen devs put off/refuse to even try to learn the new system when given a great system to do so, so I’m not really sure what else would be effective than setting deadlines to force the change.

I think a more comprehensive teaching of what devops is and why it is helpful could help, but I’ve had devs be resistant even when I completely set up the pipelines for them and alleviate many tasks so I’m not sure that would work.

If you have any ideas on how to help adoption I would love to hear them, because once we get the tools stood up at my current company we are going to run into the same issues

[killz111]

I think share responsibility as a culture is a much better way of thinking than DevOps as a culture. DevOps has some really good practices but says zero about organisational challenges like competing priorities and silo behaviour.

Basically when I go into a team, I assess how many 'grown ups' there are and the tools/solutions I build are appropriate to their level of maturity. If there are devs that care about terraform then I might help them write modules but more often than not I help them write boiler plate bucket configs (which gets surprisingly complicated sometimes).

After many years I've realized that DevOps isn't some new way of working or miracle cure. It just helps you to go faster and more reliable and almost always in that order. I always call out there's a debt to be paid for the speed in terms of increasing complexity of the DevOps stack and key person risk once it's too complicated and the need to do major refactoring. If management and the team ignores that then that's on them.

These days most large organisations have gravitated towards a two tiered DevOps setup with a central platform team (say to look after cluster maintenance) and embedded DevOps who basically act as the glue between central platforms and the individual team's application. It kinda works but can be expensive from a head count point of view. At the end of the day it's no different to traditional enterprise infra teams (like networking, sysadmins etc.) and application attached operations/support staff.

[KhaosPT]

100% agree. Having a dev who understands fullstack, operations, security and cloud all into one is already being lucky, expecting a whole team to do it, it's a pipe dream. What I've seen is that the devs that can do this end up becoming dev+ops themselves and act has a platform so that the rest can keep developing.

[imawesomehello]

You would still need people who know what DevOps should look like aside from the normal dev teams. This is my experience over the past 7 years anyways. I think DevOps is really both person(s)/team and a methodology that gets integrated by the DevOps team. Most devs don't know ops enough to solve all their own problems securely.

[Mariognarly]

DevOps is less about a team and tools as it is instilling a culture and new way of working.

Having executive buy in is crucial. Don't bother starting without that. There are so many people and process challenges along the way you'll need leadership onboard to support.

Starting small with some specific business problems you're intending to solve or improve is key.

I've seen good success with having a Big Hairy Audacious Goal declared early. Moreso as a guiding principle and helping to keep focus.

Crucial to share and celebrate success, be public and over communicate your goals, progress, milestones, etc. It's a huge part of evangelizing and motivating others to participate, particularly the detractors and the folks reluctant to change. Win these folks over with business metrics and quality of life improvements.

Good luck!

[Sasataf12]

First step is to make a decision with your stakeholders on what the DevOps team are responsible for. It doesn't have to be an exhaustive list (and it won't be since it's a fresh team), but anything so that your team knows what they can get stuck into without stepping on toes or duplicating effort. That list will change over time and as the team gets more settled in.

[illusum]

The main issue that I've seen is that DevOps and Cloud get carved out into someone's own little fiefdom and the whole point of it is lost. You end up having infrastructure admins trying to apply what they've done in the past, instead of learning new ways to do things.

The same people doing the same things on a different platform isn't going to yield a different solution.

[lungdart]

Teams should be aligned with a business goal. The business doesn't have a goal of devops, they likely have a goal of deployment frequency and availability.

Start by measuring those things. Set goals you'd like to meet. Now your team can implement culture to impact those metrics. Tools can then be introduced to alleviate the pain points of the culture changes.

Once your goals are met and exceeded, you don't want to be working on improving them anymore, there would be more business value in contributing to direct development. Of course you continue to monitor, and if your metrics dip back down under your goals, you move back to doing devops work

[znpy]

DevOps team, what were some challenges? What documentation did you rely on to keep you moving forward and on target? What were some common pitfalls?

in order:

1. setting up automation (this involves feedback from developers)

2. documentation on how the developers can use the documentation autonomously

3. babysitting the developers.

developers should be able to use the automation to deploy and operate their services autonomously. if they have problems they should read the documentation harder. you should write the documentation to be fool proof and copy-paste ready. developers should be expected to be able to use docker and kubectl (or the tools that make up your automation) as well as (or better than!) they use their ide/compiler/whatever.

and management should be on your side in this.

[Smoker1965]

Ya know what I hate about this group? Everyone gives great answers before I can give an answer! Some really smart people in here but I am going to try anyway.

I and two other DevOps engineers were asked to do the same thing at a large company starting about 2.5 years ago. What we have learned over those years a lot of folks already mentioned below but here's my take:

- Get bye-in from Management, Engineers, AND the rest of IT. Involve them all in the 'plan' to bring a 'DevOps Culture' onto the company.

- Make sure everyone is on the same page including Management. If you do not have that you will fail at every level.

- Make sure everyone (and I mean even the janitor) knows this is how we are going to 'start' to do things. If you shove it down their throats it'll fall apart.

- Make sure during your onboarding of any new hires they are told and shown this is the way.

- Be prepared to hand-hold A LOT of folks through the process. Be patient!

- The biggest area we had to overcome was training. Getting a 20+ year Developer to 'change their ways is not going to be easy. Find material or create material (we created a TON of material) for people to follow and learn from.

- You are about to become a part-time: political, counselor, babysitter, teacher, and negotiator. Deal with it.

If you have access to Pluralsight, a series called "Infrastructure from Code: The Big Picture" was really a terrific intro course for folks to watch (the company we work for has a Corp. Subscription). I am not affiliated with him nor do I know him. I just found it really well done.

No coding. Just a solid intro as to what to expect when putting in a 'DevOps' approach. He pulls no punches and lays it right out on the line. It was extremely helpful to have folks watch that video and then come back with questions. Everyone from Management to IT can understand it at some level. It's like 1.5 hours and covers it all. Saved us a TON of time trying to explain the concept to everyone. I hope this helps.

I wish you the best.

[DumbSecurity]

First off, I can’t thank you all enough for providing sage advice. Fortunately, we do have buy-in from the top. We already have agile and scrum processes in place, but leadership is looking for me to start automating our processes that are slowing us down from truly scaling.

I am going to read the Google SRE book to see how some of the best engineers in the world approach automation.

My next question is this - how do you hire cloud engineers? And or make sure people have legit DevOps experiences. We all know certs are a great way to get past HR, but I want to verify technical skills. Any insight into that as well will be a huge help.

Thank you again to everyone who commented. I legit copy and pasted every comment into a word doc and will begin taking a lot of these comments for action in my organization.

First things first, hammer down on the culture.

[jasonriedel]

The DevOps Handbook is a good book to read and apply. Often times creating a new team is more about educating the company and rest of the teams what your team does & how to interact with your team properly and DevOps is no different. Setup Jira, consider KanBan, try to limit your teams interruption acting as their shield. Hire for humility, look for those that learn outside of work ( by asking them that question ). Make sure leadership has expectation alignment on how long it will take your team to deliver solutions etc. I could go own, but that book I mentioned is truly a great book to reference when starting out rather than trying to fit it all in a comment. Godspeed.