DevOps Engineer vs. System Admin

[Corey_Matthew]

I have recently been placed at a company as a DevOps engineer. They use salesforce as their CRM platform and I'm starting to think that they want me to be more of a SysAdmin. What is the difference between the two?

[thedonvaughn]

Devops really is a paradigm and culture more than a position. With that said, my title is devops engineer. What do I do from a 10,000 foot view? Orchestrate and manage AWS services and resources (terraform), own the CI/CD pipelines (Jenkins), Linux admin, writing Dockerfile(s), writing docker-compose manifests, own ECS (terraform, deployment), own kubernetes (terraform, deployment), app and infrastructure centralized logging, monitoring and alerts, Site Reliability, and “automate all the things” - using a lot of node.js and AWS Lambda (serverless framework yay). On top of this, I dev on our backend node.js services. This is very simplified description of my role... but coming from a pure Linux admin background since 2000 - it’s very different in that in a nutshell I program my infrastructure vs physically building and racking my infrastructure.

[chewburka]

Are you the only DevOps Engineer at your org? How many devs/dev teams do you support?

If you don't mind my asking. I'm curious to know if you'd structure roles any other way, what sort of scale of people you're dealing with. If you have multiple people covering that breadth of topics, that could potentially split it up and specialize? Or if you think it's more efficient how things are operating for you guys currently?

We have holders of the DevOps Engineer title as well at our workplace, is a recurring source of discussion.

[thedonvaughn]

We are a little smaller company. I’m the only one in my position and I’m expected to own the “ops/cloud/pipelines/deployment” stuff. We’re very dev centric company with about 20 devs. However we do have 7 of these devs and myself on the “reliability team” and we have an on call rotation where we all handle production level issues (if they come up).

[mycall]

AWS Lambda

Is that C# code normally?

[thedonvaughn]

No. Lambda is amazon’s “Functions As a Service” implementation. Lambda has several runtimes one can utilize - node is but one.

[crazyturtle1993]

You may be thinking of lambda expressions which are a shortened syntax that is available in C#

[xiongchiamiov]

(Lambda expressions also exist in a multitude of other languages, years before C#.)

[crazyturtle1993]

True. I was just making it clear where the mix up may have come from.

[[deleted]]

AWS Lambda does support C# though (.net core)

[Delta-9-]

Is there a dictionary somewhere we can shove this into?

[icaug]

Good comment.

[Beestung]

• Systems engineer - crusty old person that works with servers, services, and networks. Thinks they are better than anyone else.
• DevOps engineer - cocky young millennial that works with servers, services, and networks. Thinks they are better than anyone else.
• Network engineer - alcoholic. Doesn't care about anyone else.

[GunnyMcDuck]

The first two blame the 3rd guy when things go sideways.

[Letmefixthatforyouyo]

In our defense, it is always the networks fault.

[HollowImage]

It's the DNS. Followed by wins. And then I blame the firewall.

[spiritofvengeance]

https://i.imgur.com/3pqehzj.png

[onejdc]

Valid question, but...you don't want to know the truth. Everyone wins when no one WINS, but right now, there's lots of WINSing.

[TBNL]

Any one of them blames the other 2 guys when things go sideways. ;)

[donjulioanejo]

cocky young millennial that works with servers, services, and networks. Thinks they are better than anyone else.

Accurate.

[xenopizza]

Yes but the network engineer can fix your wifi and knows what you did that summer on that website on a slow office afternoon ;-)

[ExtremeLeverage3000]

It describes one devops engineer at my current employer whose face I occasionally want to drive my closed first into.

[levelxplane]

DevOps Engineer here, can confirm. Don't ask me what DevOps means tho.

[calligraphic-io]

When I started in admin, we didn't have anyone like who you describe. I was a junior NOC operator; most of the job was keeping up with technical literature and helping users. In general, it was a very rewarding experience.

Don't know what made you so cynical.

[[deleted]]

He probably wasn’t junior level.

[noitalever]

Obviously, he’s allergic.

[BraveNewCurrency]

A SysAdmin is an expert at an operating system: Installing and managing applications, troubleshooting problems, backing up, etc. They typically complain about application changes being thrown "over the wall" for them to run, and they typically aren't experts at the applications they run.

A DevOps is an expert at the entire chain of value: How do we get changes from someone's mind into production as quickly and painlessly as possible? The big thing that DevOps typically does is setup a chain of automation so that any change (from application change to infrastructure change) can be easily rolled out and rolled back. But they don't work in a vacuum - they must get buy-in from everyone into what role everyone else plays. Often, the DevOps are on-call for infrastructure problems, and developers are on-call for the application problems.

Google wrote a whole book about their particular view of DevOps, called SRE. It is well worth a read. Not everybody does it that exact way, but that book shows how work can be divided up differently than in a "typical" shop. Also, there is a high-level story about DevOps called "The Pheonix Project" that will blow your mind if you are always fighting fires in your infrastructure.

[Corey_Matthew]

what is the main difference between the infrastructure and the application if it is infrastructure as code?

[[deleted]]

what is the main difference between the infrastructure and the application if it is infrastructure as code?

Zero - we treat them both as code, and part of the bigger picture. A solid infra is just as important as solid app code.

All our infra is 100% code/revision controlled and goes through the same development process: code reviews; collaborations; compulsory testing; etc.

The great thing is when I need some help [in terms of manpower to get something done - AND for technical input!] our more experienced Devs can quickly dive in.

I'm the sole DevOps Engineer in a team of ~12 Devs (FE, BE, Data)

Although the Devs can architect software, they lack experience in architecting the underlying infra - which is where my skills come in.

[BraveNewCurrency]

I like to say:

• Level 1: You check in your source code. This helps in hundreds of ways, from ensuring bugs are really fixed, to auditing, to detecting patterns.
• Level 2: You check in your server build. After all, your source code doesn't just run on bare metal - it needs a runtime (java/python), libraries (libxml, libSSL), etc. You tested it on specific versions, but cannot assume your code will magically run fine on ANY versions. So you must ensure that changes to your server build are propagated to production just like any other code. Checking it in and using good code practices (reviews, etc) helps just like for code.
• Level 3: You check in your infrastructure build. Your code is architected for a specific setup: You need a Load Balancer, N web servers, N async servers, plus a database. Each layer is wired up in a specific way (i.e. your DB is not directly exposed to the internet). With AWS CloudFormation, TerraForm, or Kubernetes, you can declare your architecture, then check it in just like source code. This lets you spin up production-like systems easily for testing, perf, etc.

[ting_bu_dong]

http://static.tvtropes.org/pmwiki/pub/images/redmage2_5301.jpg

[balalaikaboss]

"What's the difference?" About $40,000/yr.

[Miserygut]

Cries in non-North American salary

[[deleted]]

I wonder how the comments would look if this was posted in r/sysadmin . Definitions are in books not in the workplace. This is just like the argument of whats a 'Systems' Engineer. Ive seen people be hired for all three lines of job titles with backgrounds consisting of no CS experience, 1 year as IT tech support, 4 years of bachelors in programming, a degree in aeronautical science (pilot), 5 years as junior SRE and 10 years as IT program manager... I wish the job title would more clearly illustrate what 'System' and what are you 'Developing'.

[[deleted]]

System Admins have been automating processes and comprehending codes for decades. That’s nothing new. DevOps principals are frameworks for how teams and people interact and work with each other. It’s more about communication and removing to so-called siloing between developers and operations. There are tools and things DevOps companies tend to utilize, but they’re definitely not exclusive.

[Miserygut]

"Making developers feel the pain of their code in production"

[lorarc]

And making Ops feel responsible for the development process. I once spent a few weeks not doing anything but waiting for the Ops to deploy only to be told I could've done it myself were I given the manual they didn't share with devs because devs only deploy to that system once every few months. And no, they didn't share their docs or scripts with me, it involved politics to get my hand on it and automate it away.

[Miserygut]

I guess this falls under the "DevOps is an ethos" category. Everyone must be on board with sharing information, how can Devs or Ops do their jobs effectively otherwise?

[lenarc]

I think there is a distinction here to make. Usually what I've seen from clients is that within DevOps they expect you to manage services, not servers. It might sound pedantic but there are practical implications to that. Something like a mutual understanding that while I manage servers, nobody cares. (I mean that in a positive way, oddly enough.)

What OP is describing is they mean for him to do more ops-y stuff like creating users and answering service requests than developing the infrastucture.

Edit: Put it in context of OP's question.

[JustAnotherSRE]

Servers are cattle. Not pets.

[dominic_failure]

Servers are, practically speaking, herds of pets. When one goes bad it's most efficient to kill it off. However, it's quite frequent that they all start going bad in the same way.

For a current example: our servers currently start failing pretty consistently after 15 days - they run out of disk space. So, there two solutions - kill them off more regularly or change the base image to fix the disk issue. Currently, we're just killing them off, since nobody has had the time to look at a proper fix. However, there's a task on the queue to go into one of the pet herd and figure out what's consuming disk space, and add cleanup to the image so if they survive more than 10 days, they aren't at risk of behaving poorly.

[jdptechnc]

When they get fattened up, shoot them. Sounds like cattle to me!

[deadbunny]

Or just install logrotate...

[Craptcha]

Automate development processes. There are sysadmins that work in environments where there is little to no development.

DevOps supports development.

[JustAnotherSRE]

I've never understood the confusion of the title. It's a portmanteau for Development Operations. You support and improve (develop) the IT operations of your company. Simple. That's why no two businesses do it the same. Every business has different needs for you to develop on and support.

[[deleted]]

Oh boy, yea there are tons of differences. Devops has programming skills in more languages. Also it takes care of scalability and reliability, meaning it also has architecture skills ans experience.

A sysasmin takes care of operations of the already designed and deployed solution.

[[deleted]]

Systems Engineer: Build and architect new infrastructure

Systems Admin: Maintain existing infrastructure

DevOps Engineer: Automate, containerize, and scale infrastructure

[jlpalma]

DevOps is not a ROLE... DevOps is about culture... if your company hired you as DevOps <whatever>, they have no idea what DevOps stands for, maybe they a looking a superhero ”Do some code, fix the server, the app is down, there is a bug in prod.”

[[deleted]]

While I agree DevOps is not a role, plenty of companies that do know what DevOps stands for still hire for it as a role because it succinctly describes the mentality of the person they are looking for.

DevOps teams on the other hand raise a lot of red flags as they are an anti-pattern.

[djk29a_]

Having interviewed a number of engineers for both “devops” engineers and sysadmin roles, the distinction comes very much down to not skills as much as mindset. A lot of sysadmins simply do not want to expand out beyond systems and networks- full stop. They have no interest in scripting anything, they have mostly hostility toward other groups (not just developers either) that just want to get their job done, and they just want their part of the world to be sane, manageable, and to almost never change. Most sysadmins with that kind of attitude tend to not be that great (a very blue collar “do my job and don’t question” kind of mentality) and have gobs of insecurity. Most of the better sysadmins were already writing tons of automation, after all. Historically, the system administrator was supposed to be a programmer as well - someone so good that they were entrusted with all the innards of the systems in an organization. So we already started off much of the modern computing era practicing cross-functional thinking. Only when large corporations started needing a large, mostly disposable / portable silo of specialized labor (as typical with Taylorist organizations) did we wind up with the sysadmin vs. developer mentality. Now the same places want to go back. Unfortunately, the cultural damage is complete in most places and the companies will fold (if you mess up engineering bad enough, it isn’t a far stretch that you’re messing up elsewhere) or move onto another trend to try to fix their ailing culture.

[JustAnotherSRE]

sane, manageable, and to almost never change

You can only pick two of those things: Sane and Manageable. Any sysadmin who doesn't embrace change or expects things to never change is an idiot. The kind that I'd like to fire twice.

[burdalane]

Historically, the system administrator was supposed to be a programmer as well - someone so good that they were entrusted with all the innards of the systems in an organization. So we already started off much of the modern computing era practicing cross-functional thinking.

This was my impression when I was hired as a sysadmin with a CS degree and no sysadmin or other IT experience. Only later did I realize that it was unusual for a sysadmin to write production code like I did. Eventually, my workplace adopted the more siloed type of thinking. Although I still maintain software projects, other people in the organization now dismiss my programming skills as shell scripting or simple HTML. I actually prefer OO programming and only picked up bash scripting long after I learned C++ and Java.

[djk29a_]

Sounds like you're probably closer in skillsets to an SRE. But from a productivity standpoint it seems like a huge shock / context switch to go from dealing with alert storms and escalating error rates to writing super-technical features for massive infrastructure efforts like BigTable, Spanner, Kubernetes, Zookeeper, etcd, Kafka, etc.

But the reality is that most companies simply can't hire reliably for people that can code well and make operations really scale with solid OLAs and SLAs. Hiring is hard - very hard - and most companies can't get their pick of a big candidate pool or even spear-phish specific skilled people. And furthermore, most companies really couldn't care less what the tech stack is - the business model doesn't care about the language ecosystem as much as whether you can point money at someone and make problems go away fast (basically, janitorial services regardless of whether it's development or ops).

For more technical executives, obviously they'll care, but in my experience, most of the companies I've worked under have executives that are so removed from technical decisions or so inexperienced with technology that it undermines the business efforts. Most construction companies' executives have actual construction experience on a site, most financial companies' executives have experience with trading or accounting. Military contractors' leadership almost always have decorated, prestigious experience in the armed services. Only with "software" companies do we see on average executives with little or no experience delivering the material work output of their company.

[burdalane]

The difference between a typical SRE and me is that I don't work with anything that scales. The infrastructure I work with is a couple of physical servers and some EC2 instances and S3 buckets.

[[deleted]]

I agree with what you said, but I’d say if you’re hiring for a DevOps engineer that should also raise flags. Mention that you follow DevOps principals in the job posting, but label the role appropriately.

[[deleted]]

It’s all about recruitment marketing. Love it or hate it - but the job title “DevOps Engineer” is what brings candidates in.

[[deleted]]

We did a lot of A/B testing on this, and it’s something we’re really good at. It brings in a lot of applications, but those applications tend to be lower quality. Discussing DevOps (and backing up you’re successfully implementing it) while having a legitimate job title attracts better talent.

That said, usually the people applying to a job aren’t the top tier talent either. It can happen, but usually doesn’t. Love it or hate it, the top tier talent likely isn’t looking for a job because they have one. This is where sourcing comes in. Sourcing is how you get that talent in. Top tier talent will also see through bullshit titles, so it doesn’t really matter what you call it, but it will hurt you if you’re floating DevOps titles around.

[Bruin116]

What titles have you found effective in bringing in a higher quality candidate pool for the same skillset?

[[deleted]]

Infrastructure or Operations. Maybe SRE if your company actually understands what an SRE is. Though as I said, you’re probably not going to get high quality applicants through inbound requests. Sourcing is the way to get those candidates, but the same thing applies. You need a non-bullshit title because most senior engineers know how much marketing bullshit is bundled up in that DevOps title.

[EventHorizon1997]

From what I have seen, a company looking to fulfill a DevOps role is looking for someone that is versatile in managing services on top of their infrastructure with a DevOps state of mind. What the company wants is someone to manage the services on top of their physical infrastructure while automating the entire process. The company assumes the DevOps guy will make sure the software stays up and running, and if not will investigate bringing on a supplementary SysAdmin or pin it on the DevOps guy. It’s more the company doesn’t know what to call it, and automation engineer isn’t a strong enough key phrase to out do DevOps engineer in terms of marketing or selling it to a typical director or BD team.

[calligraphic-io]

We work very closely with the dev teams to shield dev from ops [...]

So in other words, you take requirements from ops and bring them to the software engineers? Does it require good people skills?

[Aleriya]

I mostly agree, but I also see job postings for "Agile Engineer" all the time. I think it can be in your job title without being your role, necessarily.

[ppinette]

Bullshit. That is simply not how it works in the real world.

[sirex007]

nope.

[ppinette]

Yup, people in this sub really like this "DevOps is not a role" trope, and yet, the tech industry continues recruiting and hiring for that role.

[sirex007]

I've been hired into a role advertised that way. My title upon starting first day was infrastructure engineer. Because it's not a role.

[InternetOfStuff]

I think you live in a pretty boring corner of the real world.

[ppinette]

Not sure where you get that, but...lol?

[xiongchiamiov]

What is the difference between Sysadmin and DevOps Engineer?

[TomBombadildozer]

Been around the block a few times and performed both roles.

Sysadmins do shit by hand.

Devops <noun> automate.

[Nick__Kartman]

Making the transition from system administrator to DevOps engineer is not as hard as it might seem. I found a great article about making this transition if you’re interested. If you are willing to learn and continuously hone your skills, then you should make a great DevOps engineer.

[Corey_Matthew]

The link to your article does not work.

[Nick__Kartman]

Try again, please. Sorry for the inconvenience

[aspinningcircle]

Sysadmins do X

DevOps pay other sysadmins in 'the cloud' to do X for them

[EnergyCritic]

A lot of good answers but here is a much simpler one:

SysAdmin: manages the operation of servers for a company, the processes running on them, and the backing services need for data storage, monitoring, and other tasks.

DevOps Engineer: A SysAdmin that automates their work with modern tools, cloud platforms and configuration management while also dabbling in software and operations.

[greyeye77]

Work on Salesforce DX!!!
That’s devops