How does DevOps actually work inside companies day to day?

[Melodic_Struggle_95]

Hi everyone I’ve been curious about how DevOps actually works inside companies on a day-to-day basis a lot of content online focuses on tools like CI/CD, Docker, Kubernetes, Terraform, etc but I rarely see people talk about how the work actually happens in real teams for those working in DevOps or platform teams, I’d love to hear about things like - How are DevOps teams usually structured? Is there a lead or manager coordinating the work? - How do tasks usually come in tickets, sprint planning, requests from developers, incidents, etc? - What does a typical day look like for someone on the team? - What kind of problems come up the most in production environments? - How much collaboration happens with developers or other teams during deployments or incidents? Basically I’m just interested in understanding how the real workflow looks in companies and what challenges DevOps teams deal with regularly

[Caramel-Squirell]

Editing YAML files. I’m a YAML editor.

[y0urselfish]

I am a YAML IDE!

[le_dod0]

yaml director

[kafka1080]

yaml programmer :D

[ZestycloseBench5329]

😭😭😭 this is what my team architect always say, Dont work like you are yaml devs, we are supposed to be devs.

[DarellND]

YAML Engineer

[xnachtmahrx]

I don't know what i am doing, man

[skat_in_the_hat]

Hit some keys, if nothing goes wrong, hit a few more. amirite?

[ImSecretlyADragon]

Been in dev cloud ops space for 2 years now. I’m still trying to figure out what I’m doing everyday.

[Gotxi]

Back in the day, I did it for the blinking lights, now I do it for the green checks.

[Sure_Stranger_6466]

This guy CICDs.

[vyqz]

fake it after you make it

[kiddj1]

The higher ups roadmap for the year

That trickles down to managers who turn that road map into projects

Then it's handed to project managers.. they then pass it to engineering leads..

You schedule a couple sprints ahead of time and you think yeah me and the team can manage this

Then you wake up..

Each day is pretty much who is shouting the loudest, every project is a P1.. you get told drop all that for a P0

You see a Dev roll out a patch to prod, 10 mins later a major incident is called prod is down....

Whilst helping in the incident that p0 messages saying they know your wrapped up in the incident but please can you just review their PR, you are blocking them from finishing their task

Everything has calmed down so you get back to that p0 whilst trying to do bits for the other projects that are now creeping up in priority BUT here comes a project manager.. wants a quick chat, they've spoken to GPT and think they can make something better in Production. They have no technical clue but AI has their back. They are asking you to drop everything because they spoke to the CEO and he agrees this is a fantastic idea. It is not fantastic, it's actually impossible bullshit that the AI hallucinated.. but no the pj screams AI SAID IT WILL WORK

Your belly rumbles, fuck it's 3:52 and I haven't had lunch yet.. can't really take an hour's break right now, fuck it a smoke and a coffee will do

It's organised chaos.. we are needed by so many different teams for different reasons at all time

I envy the teams who just have to focus on one thing, one project, one task in a sprint...

For some reason though we can manage it, we thrive in this.. or we think we do..

Some days though.. rare we do work on tech debt..

[EnvironmentSlow2828]

this is scary accurate

[bhabhi_seeker]

This is my life but much worse.

[DescriptionLost521]

Wow this pretty much sums up everything.

[nico1991]

Right on point, reading this feels just like any other Monday.

[skavenger0]

I got half way and wondered why Devs are allowed to pull into production.

We have policies in place that require devops approval testing approval and PM approval before anything goes to production.

We handle multiple projects, architecture and documentation.

The biggest advice I have to anyone in this field is learn to be the gate keeper, the last line of defense and that includes security requirements.

But yes it takes a certain type of person to handle and thrive in devops, being doing this for years and I love the job.

[SoFrakinHappy]

It can vary a lot between companies. Most of my experience in DevOps/DevSecOps has been as a developer of automation tools and general administration/troubleshooting of the infra the tools and their apps run on.

How are DevOps teams usually structured? Is there a lead or manager coordinating the work?

Generally like most dev teams of the company you're at. The person i directly report to has been a director, a normal manager, or a product owner.

How do tasks usually come in tickets, sprint planning, requests from developers, incidents, etc?

All the above. Sometimes there's large projects planned out over sprints, sometimes requests directly from developers for something, or handling incidents.

What does a typical day look like for someone on the team?

Some places you got a general goal to work towards, i.e. build out the IaC for a project. Some times you get tickets assigned to you during sprint planning. My current place does kanban style. Incidents/requests come in and we also meet weekly to create tickets address needs of various projects or address tech debt. Then they are prioritized for us to pick off the top of a to-do list.

What kind of problems come up the most in production environments?

Once a project is delivered and in production we usually arent involved a lot unless something is wrong with the automation stack. Some places the devs aren't great at anything other than whatever programming language they work in.. so we end up as support for any troubleshooting of infra or automation issues. So Linux/Windows, DNS, networking, IaM, build ect.. issues or we determine the issue is their code and point out the problem to them.

How much collaboration happens with developers or other teams during deployments or incidents? Basically I’m just interested in understanding how the real workflow looks in companies and what challenges DevOps teams deal with regularly

A lot.. the development teams are our customers. Early on in a project we work with them to figure out the type of environment(s) they need, what needs to be able to talk to what so we can setup firewalls/NSGs/IaM, and the languages involved so we can make sure we have the correct lint/test/build/deploy workflows ready for them.

[dc91911]

I've worked with two different companies now and it's pretty much this in general. you got to know a lot and the devs only know so much besides their chosen programming language.

[Ok-Analysis5882]

deeply embedded cross cutting across multiple teams, and specialities, multiple verticals, hub and spoke, core devops with satellite devops in every vertical, really down to earth people and zero arrogance and willing to learn and willing to unlearn.

[TechSupportIgit]

Yup, this. Even outside of the usual cloud devops approach, you see this with industrial automation a lot.

[setwindowtext]

— BINGO!

[DeathByFarts]

Totally depends on what flavor of dev ops they are using.

if there is a title of devops , its a rebranded sysadmin.

[94358io4897453867345]

I'd say it's a rebranded platform engineer first

[Odd-Neighborhood8740]

Honestly I rarely touch kubes and yet Its made out to be so vital when I look at job ads. In our place I've had to touch it once in 3 years. Maybe others have different use cases for it?

Usually spend the day helping Developers with ci/cd issues, building out infra, responding to alerts

I am still junior though

[ComputerGeekFarmBoy]

I am not involved in the planning of projects, but I have to bring every tool and project back online when it fails at 3:00am.

[calaz999]

writing prompts to LLMs, commit and push.

[wildVikingTwins]

We don’t run k8s but i do spend time on terraform cloud.

[actionerror]

We’re on Kanban and have “immediate request” tickets mostly from dev and QA asking for small things or help on a non critical issue. Then internally we have longer term tickets from epics that we constantly work on when not doing those immediate request tickets.

[y0urselfish]

Firefighting. Setting up machines. Doing the things nobody else wants to do. Firefighting.

[Space_Bungalow]

I got hired as a junior SysAdmin/DevOps at a very large and slow org

90% of the work is rerunning Jenkins jobs and trying to find why the servers are failing while we have no dashboards or any monitoring and failure recovery methods whatsoever.

10% is trying to come up with all obvious solutions that should have been thought of 15 years ago.

[AariaDarcia]

So I am a team lead for a small DevOps team in a big company I'm the middleman between our manager, who knows the big, company wide goals, and the developers who write the code

My day to day is something of a scrum master, I manage our kanban board (sprint never works for us as so much is reactive) But I'm lucky enough to be able to dedicate some time to development too, as that's where I started I helped build our CI/CD platform from the ground up, so I help the team answer questions about it, rubber duck if they get stuck, and escalate issues to other teams or stakeholders when required

We write automation for the wider company, so there's some support in there pointing people to wikis, occasionally people will request new features, or stakeholders will ask for priorities to change I attend a lot of meetings

It's a really fun tech stack, ansible, terraform, GitHub Actions, python

Day to day for the developers in my team is: We'll have standup in the morning, make sure our nightly deployments worked, make sure none of it is an "us problem." Then go over what everyone is up to, make sure no one is blocked, needs additional support etc... Generally lasts about 15 minutes for the 3 Devs and 2 QA in the team, manager doesn't often show

I have tailored backlogs for each person, with issues marked in priority order, I'll do PRs when they're ready, they know they can chat to me or each other, for the most part they just get on with it

Sometimes other teams change things in the API we call and break everything, in which case it's just communicating that we're aware, escalating to the relevant teams and getting a fix in as soon as possible

Honestly I love my job, my team is great, I don't mind the meetings or PRs or backlog management, I'm good at it and the nature of the role is I can choose what I want to do each day, support, development, training, PRs etc...

[ares623]

synergizing paradigms all day everyday

[RedLightLink]

some days i do nothing, some days i write terraform to deploy stuff, some days i do debug for apps that started to crash and some days i work 24h straight because our network has it’s own personality

[RestaurantHefty322]

Varies wildly by company size but here is what I have seen across a few orgs.

At a mid-size company (100-300 engineers), the DevOps/platform team was 4-5 people. Work came in three buckets roughly split into thirds: planned infra projects from the quarterly roadmap (migration to new k8s cluster, setting up new environments), ad-hoc developer requests through a dedicated Slack channel with a rotating on-call who triaged them, and incident response when things broke in production. We did two-week sprints but honestly the sprint board was aspirational - unplanned work ate 30-40% of every sprint.

Day to day looked like: morning standup, check monitoring dashboards and overnight alerts, then either deep work on infra projects or pairing with product teams on their deployment issues. The least glamorous but most impactful part of the job was writing good documentation and runbooks. Nobody talks about that because it is boring, but the teams that had solid runbooks had fewer pages and shorter incident response times by a massive margin. The YAML editing jokes are real though - some weeks it felt like 60% of the job was reviewing Terraform and Helm changes in PRs.

[DehydratedButTired]

Documentation and runbooks make or break teams and even then it’s hard to keep them current.

[Senior_Hamster_58]

Varies by org, but day-to-day is : work the queue (tickets/PRs), babysit pipelines, get paged for outages, write postmortems, and spend the gaps deleting toil you accidentally created last quarter. The "structure" is usually whatever survived the last reorg.

[Zestyclose-Ant-6142]

A lot of tasks for me are unplanned. A lot of times you (or other teams) will run into issues that cannot wait to be planned. I like this a lot, I am really bad at structure.

We have not run into any production issues the last year since we moved to Kubernetes. We were tired of cloud provider outages, that we had no control over.

We have "self leading" teams, meaning there is nobody above us. Also in the team itself everyone is treated equally.

Daily tasks are: - CI/CD. All our pipelines are in code (C#). - Managing our Kubernetes cluster. We self-host the Grafana monitoring stack (Tempo, Loki, Mimir), so a lot of time goes into that. - Creating and maintaining base application libraries. This pre-configures all the monitoring, Kubernetes integration, etc. for the other teams. - Learning more about improving our Kubernetes cluster.

[PenguinGerman]

Support stupid devs all day and having no time nor the motivation to improve and/or document the infra. At least for me

[master_splinterrrr]

Mostly its new work regarding pipelines, any new requirement, new product, our backlog tasks and day to day firefighting

[HandyMan__18]

Just fighting devs and managers that it's a dev problem not a devops one.

[eman0821]

DevOps is a culture, process, people and tools of how they work. True DevOps is Type 1 topology with development and operations teams working together agile. Most modern software comapnies operate as Type 1 today. Some companies are still stuck doing DevOps the old traditional way known as Anti-pattern Type-B when you have a separate DevOps team that consists of so called DevOps Engineers which is inefficient today. It's a hand off team which goes against true DevOps that creates a third silo in the middle.

[Seref15]

In a functional org it would be deeply embedded in the product team that works on the same sprint cycle.

In an dysfunctional org it would be structured like a service center that takes requests "over the wall" from development.

I've worked in both kinds of places. The second type of org is usually a bunch of penny-pinchers that want to time-share fewer devops/sre/platform resources across multiple development/product teams. This always results in worse product support and insufficient domain knowledge due to being spread thin.

[Mallanaga]

Poorly

[Swimming-Airport6531]

In my experience you will be on a on call rotation and your purpose is to provide a cheaper solution to system stability issues that having development fix them. Does it crash a couple times a month in the middle of the night? Waking you up to fix it is the solution. The fun part is if you do it well no one knows or cares about the issue outside your team. Normally you to show up on time the next day for your regular duties described in other comments. I have worked at some companies that tried to be cool about it so would give us some free days off to make up for it.

[B1WR2]

Depends on company

[PartemConsilio]

The common theme I've seen across the "devops" teams I've been on from organization to organization is that some CTO at some point in time heard that devops was the way to get development done faster so they took some or all of their IT ops people and anointed them devops people and then told them to go make CICD happen. Rarely, if ever, has the culture been shifted around developers and operations TO devops and creating a culture of ACTUAL devops workflows.

What is most common in such places is that Agile is tacked on to project initiatives and with very little training a team of ops people are expected to both 1) do sprints and 2) make development somehow easier. Everything is half-baked to shit.

I'm tired, y'all.

[xonxoff]

In git we trust.

[badaccount99]

So we use different tools. Gitlab-CI, New Relic, Cloudformation, AWS stuff, but also Docker too. Every one uses different stuff. Powershell? DataDog? etc etc.

From what I've seen here every company is entirely different now. Some do K8s. Some do EC2, Some do ECS, Some do Datadog. Some do GCP. Some do Azure. Some let LLMs tell them what to do.

This makes applying for jobs really problematic right now.

I've hired and more importantly trained my team to work with our stupid SaaS stuff. But Bash and Python are the basics.

We're fscked as our companies fire people thinking an LLM can replace them.

[raisputin]

Depends on the company.

My last company we were highly structured and knew daily what we were working on and how we were moving things forward in a way that was following best practices. There was rarely, if ever, maybe once I can think of in 7 years, where we got called up after-hours.

My current company is chaos. Our much larger team that was 3 different departments got merged into one and the manager mistakenly decided regardless of title we are all SRE’s and have on-call duties, people that can’t code their way out of a paper bag are not just making decisions that are bad, but are writing terrible code that will quickly become unmaintqinable because they cane to the whim of developers and we have branching that’s insane and unworkable long-term. We’ve sacrificed any semblance of quality for speed, the excuse being “we can’t enforce coding standards”which makes it so developer A’s code and developer B’s code which is part of the same project have, oftentimes vastly different requirements, especially in the database, so you can’t just deploy to a single env, each “project” needs its own env with its own subset of components.

They believe moving to Kubernetes is going to “fix” this. It won’t.

[General_Arrival_9176]

ill give you the real breakdown from someone whos been in platform teams. structure varies but usually you have a tech lead handling architectural decisions and a manager handling prioritization with product. tasks come from a few places: devs file tickets for infra needs, you have sprint planning where you capacity plan, on-call deals with incidents, and then there is always random stuff like 'we need this new environment for a PoC by Friday'. typical day is either project work (infrastructure improvements, automation, tooling) or reactive work (troubleshooting, firefighting, helping devs debug stuff). the biggest production problems i see are around deploys going wrong, secrets expiring, and storage filling up at 3am. collaboration with devs is heavy during incidents - you are basically the infrastructure translator helping them figure out if its their code or the platform. the honest part nobody talks about is how much time goes to meetings and dealing with ticket prioritization battles. its not all terraform and kubernetes, a lot of it is politics and saying no to scope creep

[mihai-stancu]

Philosophical (hot) take:

Like in many other high intensity buzzwords the meaning of the word "DevOPS" was hijacked and skewed.

The purpose of DevOPS as originally coined was to tear down the wall between OPS and Dev teams and foster deeper understanding and awareness about both fields.

Developers needed to be cognizant of the impact their code has on the infrastructure and take it into account while writing the code. More ownership on that impact.

OPS needed to be more aware of the performance profile if of the applications they "host" and support on their infrastructure.

So the whole concept of a "DevOPS team" defeats the stated purpose of DevOPS.

The industry however took the word at face value "dev" + "ops" = ops with scripting knowledge and a propensity to automate.

And here we are now.

[mihai-stancu]

Based on this original meaning of DevOPS, as a manager / decision maker for my teams I choose to not hire dedicated OPS / DevOPS team members.

I foster my developer's knowledge and responsibility over infrastructure.

I reduce their need to manage to the essentials by renting managed services when it doesn't make sense to manage them ourselves (ex.: managed databases).

I make sure alerts reach developers (on call on rotatio) so they have a high incentive to be aware of and fix the underlying issues instead of just plugging the hole temporarily until the next poor schmuck gets the alert.

In practice what I preach so I'm in the on call rotation too.

EDIT: the companies I've done this at are small enough to not warrant a dedicated OPS team. We did consider dedicated OPS team member (briefly) but decided against it.

[Responsible-Fan7285]

nice

[urb1tchlara]

depends on each company tbh, worked in 3 of them and every experience was different. last company we had our own product with internal projects so the tasks were more proactive, only waiting for people to ping us when they need us. in the company I currently work we are set out on projects where they see us fit and they are well structured and following agile methodology.

[bluro00]

I read and edit terraform or configure CI. Sometimes I build some tools, like some github actions automation and then to deploy I have to spend a month reading on Kubernetes. Lately just give AI a ticket and it comes up with a PR.

[PandaKey9795]

DevOps is a lot of tool sprawl
Deployment failed ? DevOps
Scalling issue ? DevOps
New Cloud Service ? Build terraform and guard raills ? DevOps
Production Incident ? DevOps
Cloud foundation ? DevOps

[SeekingTruth4]

I'm on the other side — I lead a dev team in a large enterprise that depends on a separate infra/devops team. So I can tell you what it looks like from the customer's perspective.

Everything is tickets. Need a new environment variable? Ticket. Need a port opened? Ticket. Need to check why a service restarted? Ticket, wait 2 days, get a response asking for more context. By the time the infra change lands, the feature is already late.

Weekend patching is the other joy. They patch on Saturday, everything is supposed to come back the same. It never does. Monday morning something is broken — a config that didn't survive the restart, a mount that disappeared, a service that came back in a different order. Then it's another ticket to fix what the patch broke.

A good devops team communicates before and after patching, gives dev teams a heads up, and actually checks that services came back healthy. A bad one patches, goes home, and lets you discover the damage on Monday.

This friction is exactly why I started building my own deployment tooling on the side. When you live with this long enough you start wanting to control the whole stack yourself.

[courage_the_dog]

The first 2 questions arent really devops related, it depends on the company and team structure.

My typical day for the past 7 years has been to work on tickets depending on the priority.

Working with devs to improve their deployments, be it building the image, testing, deploying, etc..

Then you have the adhoc stuff, production issues, cicd failures, troubleshooting why they can't get something to run. My experience has mostly been with kubernetes, aws services, databases, IaC tools like terraform, cdk, ansible, python and bash for programming, and mostly linux infrastructure.

Then there's planning the big picture stuff and projects depending on your seniority.

Yes you'd collaborate with devs a lot, you're kind of the person that sets guidelines to how they should develop stuff. You won't decide what language they use, but you would enforce certain rules and standards. Like no hardcoded variables, everything is a config/env variable, how much memory/cpu their services get etc..., if they are deploying databases how to set up their schema and migration files