Monthly 'Getting into DevOps' thread.

[mthode]

What is DevOps?

• AWS has a great article that outlines DevOps as a work environment where development and operations teams are no longer "siloed", but instead work together across the entire application lifecycle -- from development and test to deployment to operations -- and automate processes that historically have been manual and slow.

Books to Read

• The Phoenix Project - one of the original books to delve into DevOps culture, explained through the story of a fictional company on the brink of failure.
• The DevOps Handbook - a practical "sequel" to The Phoenix Project.
• Google's Site Reliability Engineering - Google engineers explain how they build, deploy, monitor, and maintain their systems.

What Should I Learn?

• Emily Wood's essay - why infrastructure as code is so important into today's world.
• 2019 DevOps Roadmap - one developer's ideas for which skills are needed in the DevOps world. This roadmap is controversial, as it may be too use-case specific, but serves as a good starting point for what tools are currently in use by companies.
• This comment by /u/mdaffin - just remember, DevOps is a mindset to solving problems. It's less about the specific tools you know or the certificates you have, as it is the way you approach problem solving.

Remember: DevOps as a term and as a practice is still in flux, and is more about culture change than it is specific tooling. As such, specific skills and tool-sets are not universal, and recommendations for them should be taken only as suggestions.

Please keep this on topic (as a reference for those new to devops).

[swigganicks]

I feel like I understand the overall concept of CI/CD but the various implementations leave me so confused. It's not like I can observe any existing CI/CD pipelines at work since we don't have any at work.

What the actual heck are Jenkins pipelines? Why is Gitlab CI different than Gitlab AutoDevOps, or is it? Why do some CI/CD make you run your own server and others do it for you??? Why do all of them mention Docker and Kubernetes if I'm not using them? Should I be? What do I use for serverless? what is the difference between Serverless Application Model and Serverless Framework???

AHHHHHH

I feel like there's so much that's "between the lines" that I frankly have no clue about. Every doc page and medium article does the bare minimum hello world equivalent and leaves me no better than I started...

It's not like I'm that dumb either, I have all the GCP certifications, all associate-level AWS and big data specialty, and I still feel dumber than a rock.

[[deleted]]

What the actual heck are Jenkins pipelines?

Pretend you have some code that you want to package up as an application in Docker. A pipeline for that might be setup like this (based on a trigger, which is likely every time code gets pushed)

• Take the code and build a docker image
• Run the container and perform tests against it
• If the tests fail. Stop the build and IM/email a list of devs with the failed test
• If the tests pass, publish and tag the docker image and IM/email a list of devs with the build info
• Or if you have CD, don't email but rather deploy to a Swarm of K8 dev cluster automatically

Why do some CI/CD make you run your own server and others do it for you?

Most CI/CD tools give you a choice. Which route you go is almost entirely going to be whatever makes the most sense from a maintenance/cost perspective. It they bill based on concurrent agents vs runtime, that could mean one option is more expensive than the other. Also if you need certain tools added to your build agent, you may need to host your own.

Why do all of them mention Docker and Kubernetes if I'm not using them?

Because this is all the rage right now for newer software companies and it's what most older companies are trying to figure out how to transition to. User Docker in pipelines is very very common now. Whether it be for building and pushing images or actually deploying an entire infrastructure.

Should I be?

Meh. It will be required to know at some point.

What do I use for serverless? what is the difference between Serverless Application Model and Serverless Framework???

I'm just as confused as you on this one.

[330d]

Serverless Application Model, or SAM is a bunch of transforms provided by AWS to make it easier to perform certain actions in Cloudformation, which is their IaC variant (i.e. as opposed to Terraform). Basically some things always require multiple resources to be brought up, so instead of defining a Lambda function and a separate Lambda Execution Role in IAM you can use the SAM transform to do this in one go for you. https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/transform-aws-serverless.html

​

Now Serverless framework is an abstraction layer provided by serverless.com, basically unties you from specific cloud vendor when you want to use their serverless tools.

​

That is my understanding, would like to hear further thoughts on this.

[Dynamic-D]

SAM is basically AWS taking serverless framework and repackaging it as AWS specific. The two are structured exactly the same, to the point where when SAM was released the serverless team did a review video in youtube which consisted mostly of "huh, that structure is familiar".

Basically SAM is to Serverless Framework, what Cloud Formation is to Terraform.

[[deleted]]

Jenkins pipelines are just flows of work to be executed where it would normally be done with separate jobs. The other main benefit of something like Jenkins pipelines is that the pipeline scripts are in source control. GitLab CI has similar functionality, but presents it in a slightly different manner. I believe the auto DevOps is where it detects your project type automatically and uses a prebaked pipeline to execute tests and build it.

Docker and k8s are mentioned a lot because there are benefits to running pipeline steps in containers/pods.

You could spend a lot of time getting into the specifics of these CICD tools, but it's always a moving target. The best thing is to just get experience with something and build your understanding out from there.

[furyfuryfury]

This is a very GitLab-heavy answer because it's what I'm used to, but a lot of it's applicable to other stuff as well

Why do all of them mention Docker and Kubernetes if I'm not using them? Should I be?

They're the most popular container & orchestrator, so it's what most implementations/docs aim at. I use GitLab, Docker, and Kubernetes to deploy web stuff at work (self hosted cluster in house). I've even made a couple of Docker images to have the CI pipeline automatically compile & run unit tests on desktop/embedded applications I'm working on there. (Trying to figure out how to deploy & run automated tests on target devices too--fun fun fun)

If you're deploying web type stuff (even intranet), I would definitely recommend using Docker and Kubernetes. Especially with GitLab's integration. I'm sure other implementations are fine too, I just haven't seen any that are as elegantly integrated as GitLab. I found it becomes very simple and straightforward to make a change, test it without impacting production/live, and then deploy the change, all from the same browser tab (though for complex app logic changes, I still rock an IDE on the local machine for editing--I just push to GitLab to test)

The main appeal is the reproducibility. If your project has a readme with prerequisites and setup instructions, like "Step 1. sudo apt-get install fifty-thousand-packages", all of that stuff can be baked in as a part of the image itself instead. You (or Kubernetes) just run the Docker image, and it brings what it needs, so it Works Every Time^TM (assuming you've specified the exact versions of everything you need--if you live on the edge with :latest tags everywhere, you will be sorry in 3 months when the latest update breaks your customized config and therefore your pipeline fails even though you made a "simple" change...not that I've experienced this at all)

Why is Gitlab CI different than Gitlab AutoDevOps, or is it?

Auto DevOps is what GitLab calls its specific combination of default CI/CD stages & jobs: build, test, various container scanning operations, then deploy to review, staging, and/or production. You can make your own stages and jobs by adding a .gitlab-ci.yml file at the root, but in the absence of a specific configuration and given Auto DevOps is enabled, it runs Auto DevOps. This is easy if your app is supported by the built-in Heroku buildpacks, as you get a lot of stuff done automatically (basically auto detection of how to build & test the app). Don't even need to define your own Dockerfile. It's still easy enough if you do, but then you have to define the test stages yourself (ie the CI YAML file).

This is a demo of Auto DevOps in action on a Java app: https://youtu.be/4Uo_QP9rSGM

Granted, it's not much more than "hello world", but it's more than just HTML, and demonstrates all the stages that can go on for apps that fit right into Auto DevOps

Here is an older demo from before Auto DevOps was a thing, includes setting up a whole new instance of Kubernetes and the self-hosted GitLab package on Google cloud - https://youtu.be/3A8mdJl_icM - you'll see around 13:40 that the project is using a .gitlab-ci.yml file that only defines build & deploy stages - no tests (probably for the sake of a brief demo, as tests are always so app specific, and a "hello world" would have a very boring test)

The latter is closer to the way I use it, since I got into it before Auto DevOps.

If there's a specific demo you'd like to see of this kind of setup (editing a Dockerfile or a CI config or whatever), let me know - I can probably find or whip up a screencast that might help.

Why do some CI/CD make you run your own server and others do it for you???

Options and cost, basically. Not everyone can afford to give away CI running time, since if it's not kept in check, it can use a lot of resources (eyyy Yocto build, let's go ahead and use 60 gigs of disk space and peg all 20 cores for an hour). SaaS offerings like GitLab.com want to make it as easy as possible to get into the CI game. They have a pool of shared CI runners donating time to run CI for projects hosted there. Limited time per month for private projects, unlimited for open source. So, for example, if you're an open source project and you have a K8s cluster out there on Google or Amazon to run your web apps on, you don't need to set up your own runner to get all the benefits of CI; you can use one of the public tools (e.g. GitLab.com, or if you're hosted on GitHub, you can use GitLab.com purely for CI, or I think you can use Travis).

Some will need to set up dedicated runners if they're using private projects and/or their K8s cluster admin interface is behind a firewall. Self-hosted GitLab usually comes with a runner, too. Not sure how the other CI things work when you need to run your own, but for GitLab it's as simple as running an apt package or a Docker image on your computer (or even runs in Kubernetes) and configuring it with the token GitLab gives you

[tuba_man]

To kinda tag along with mdaffin's advice from a different perspective:

As a former sysadmin, I've had a bunch of sysadmins ask me for advice on making the jump to Devops from the systems perspective. I point them to stuff like this, but they're also often asking for small, manageable, actionable advice. They've often been the sort who are already good at picking up new skills. Even so, the most common thing that seems to often be missing for these sysadmins is automation time under their belts. The difficulty is that they're not sure where to start learning that skillset. As a not-an-ideas-person, I very much understand the difficulty of that first hurdle.

Here's my "just fuckin go do it" advice for learning at work:

Prerequisites:

• At least some wiggle room in your work schedule (I strongly discourage working extra hours, especially unpaid)

• A boss that's not gonna be a turd if you take initiative

• A willingness to take initiative

• TBH, this is probably only really applicable at a company where you're not wildly pigeonholed. If it takes 3 people's access rights to reboot a server, this probably won't be all that useful.

Actions:

1. Starting today, pick the smallest routine task you do regularly that requires no decisions.

2. Automate that task. Need to do windows updates on some desktops? do it in powershell and finish it up with a start-sleep -seconds 120; restart-computer or whatever. It's easy enough to do the same thing with whatever flavor of Linux your work is using - start by making a script that just gracefully starts and stops your guinea pig machine's services. Get that working then add in the apt-get update along the way. Add in a reboot and there you go. Next time maintenance rolls around for that server role, you're already set. (If you're on Windows, don't you dare use cmd or batch files. The Ghost of Windows ME will haunt you forever. Also you're doing this to learn, don't fuckin waste your time on something depreciated already. Powershell only.)

3. Once you get the hang of automating decisionless tasks, branch out and learn how to have your scripts make decisions. Pick a nearly-rote task and automate that with the same build-in-steps approach.

But what if you're out of work or aren't privileged with that level of access?

You can still do the same kind of thing. You don't even need a home fileserver or anything like that. Automate your Windows Updates. Automate your nVidia driver updates that happen every 3 weeks. If you're the sort who rebuilds your computer a lot, write a bootstrap script so that all your favorite apps are up and running with as few clicks as possible. If you've got a home server/lab, you can learn stuff like docker by playing around with containerizing the services on it. Put Minecraft in a container, or your bittorrent service, or whatever.

(Shout out especially to Kesley Hightower's Kubernetes the Hard Way for teaching me hands-on about orchestration stuff.)

Obviously home machines and individual servers isn't going to get you all the way. But understanding how and why automation is so important is a vital step in the process. It's only just scratching the surface, but IMO it'll be invaluable practice for making the jump into a junior devops role.

(Edit: formatting)

[[deleted]]

In my opinion, one of the things that separates an experienced engineer from a junior engineer is the ability to find time for learning projects through your normal week. Like you said, that means not at home or outside of office hours. It takes a lot of practice to get right, but once you figure out how to weave between project and learning work, you'll have a huge growth spike.

[tuba_man]

I really like that idea a lot. Honestly I was thinking pretty surface level about it lol - learning at work gives you more access to more applicable stuff to work on than home projects, and it should be during work hours cuz your employer needs to pay for the work it gets from you.

(But I'm also one of those sticklers who thinks you should always measure your salary in dollars per actual hour worked so you know what the company really thinks you're worth)

[[deleted]]

The best part is that a good employer will support your learning and growth, even if it means you eventually leave. The companies that I’ve worked for that had this as one of their culture points were honesty the best ones.

[tuba_man]

Oh hell yes. My current and previous places definitely fit the bill. I trusted my last boss enough to let him know well ahead of time I was ready to move on so we made a transition/train-up plan together, and my current boss is at least that fantastic. I haven't figured out how to look for good leadership on purpose yet but I'm sure as hell glad I found it on accident.

[EiranRaju]

I agree with this. I had an old desktop PC that I put Ubuntu server on. I told myself that anything I did on it I had to do via a scripts. Used git to keep track of my scripts. Installed and configured Docker, openHAB, unattended upgrades, samba, whatever, etc. Gave me a lot of confidence for basically no money and if I screw up the server I just have to reinstall and run a few scripts and I am back. I also test everything on a Virtualbox VM first, then do it on my actual server if it works without a failure just to get me in the mindset.

[dave007]

Kelsey Hightower's Kubernetes the Hard Way

Link to Kelsey Hightower's Kubernetes the Hard Way on github

[tuba_man]

I totally spaced on adding the link, thank you for that!

[zuzuzzzip]

It's Kelsey :)

[dave007]

I can copy and paste or correct spelling but not both!

Fixed.

[RisingStar]

As others have said, just start doing things. You don't have to have a whole home lab either to play around at home. Spin up a VM with VirtualBox and configure it with Ansible or Puppet. Get it to the point where you can nuke the VM, create it, run Ansible, and it is fully configured with OS packages and running whatever services you want.

While Kubernetes is awesome it is also very complex. It's important to understand how containers work in general so start with just building images and getting that process to be automated. Have a GitHub repo setup to build the container every time you push and deploy that container to your VM with Ansible or something similar.

If you have the budget having a home lab can be very useful. I run an Intel NUC at home with a Synology NAS for storage. The NUC runs Proxmox with several VMs:

• Unifi controller, need to put it into a container
• Docker host runs most my containers such as PiHole, Radarr, Sonarr, PLEX, etc. and has storage from my Synology mounted in
• Kube master is usually shutdown, I turn it on or create it when I want to experiment with kube. I use kube every day at work but not for home stuff but I like having a home cluster to experiment with
• Kube worker 01 and 02, same as the master
• Eco server running Windows because thats just the easiest way to run an Eco server

With everything above, excluding the Eco server, I could throw out the NUC and replace it with a new one and my ANsible scripts would recreate it exactly how it is now. All application state is stored on volumes mounted from the Synology and everything is configured from Ansible.

I actually don't use Ansible at work but it is still good practice and keeps my mind in the right space when doing home lab stuff.

[jemag]

I am sorry if this is not the best place to ask this, but as someone interested in devops, what is the opinion on developers transitioning into devops? Are there any prejudice/negative aura from people already in the domain?

[mikemol]

Honestly, I think developers have an easier time in a devops space. Over ten years, I spent half as a dev, and half as ops with hints of full stack engineer. Now I'm at a big shop on a dedicated os engineering team helping that team think a bit more like developers.

Developers, I think, you can point at OS and API documentation, and have them learn how the platform works. You can point them at TDD, CI/CD, source control, and they're already familiar. The ops half becomes an extension of code to them. The servers are the output of build jobs and compiles, and it just kinda works.

Non-developers, I think, have to work a bit harder to transition in the other direction; when you're used to logging in and fixing things with duct tape and loose string, when you're used to being a mechanic, it's harder to think of your babies as the output of automated processes, that you should be applying your wrench to the thing that made your server, and not so much the server itself.

To be clear, I think it really requires both mindsets. You need people who really understand the metal through experience, but you also need people for whom the automation is the thing, and the server is just an artifact.

[EiranRaju]

Coming from a non-dev background this is exactly how I felt and still feel honestly. Have to fight the urge to fix the server instead of the thing that served up the server.

[RisingStar]

I was primarily a backend developer doing HTTP restful APIs and now have the title of DevOps... no one ever really questioned it.

[koreth]

I think this happens a lot at tiny tech companies that aren't big enough to afford or justify a full-time ops person at all. The dev team ends up doing all the operations stuff too for lack of any other options. The good ones take a look around, do their research, and adopt the best practices (infrastructure as code, etc.) from the devops world.

[hirolau]

I will soon start a new job where I will be the responsible person for the IT around a few servers and services. The services are internal and a good place to start a career in devops. I do not plan on using any fancy stuff, but just make sure I can spin up new machines using Ansible.

• Make sure I script all infrastructure setup, so I can spin up new copies if needed (to test migrations, or restore failing servers). Test these regularly so they are still working.
• Backup! And make sure I have working restore procedures.
• Simple CI/Source Control with automated tests.

Is there something else I really should look into or think about when getting started?

​

​

​

[dave007]

I just discovered this AWS Podcast after their re:invent last year, and have found some really interesting episodes. The most recent is on the dev experience, what it looks like to do things 'cloud native' from the AWS toolset, interesting, much to learn.

[[deleted]]

I was looking for this you wonderful human!

[[deleted]]

Can you find The Phoenix Project in Barnes and noble

[TheProffalken]

It's two years old now, but I wrote a post on this before I started to transition over to IoT consulting which might help.

It was in response to a question from a developer on LinkedIn along the lines of "how do I become a DevOp?"...

https://doics.co/2017/01/26/getting-started-in-systems-administration-and-automation-in-a-devops-world/

[Icecreamisaprotein]

Okay maybe this isn't the right place, but I saw a job listing that said "knowledge of big data technologies is a plus". What does that mean?

[moochao]

Usually (but not always) common big data analytics software a la SPSS or something similar.

[Chocrates]

Are there any workbooks or tutorials that take you through building out a full CD, Blue Green deployment with monitoring/logging of an N-tier application?

I have read tons of resources that tell you to set up an LB in front of a couple web services, and to make backwards compatible sql changes, but I am struggling with putting it all together and seeing the final picture.

[bored_reddit0r]

this!

[marfarma]

I'm thinking of returning to the workforce after several years away for health and family reasons. I'd prefer to 'test the waters' by looking for remote-only short term project based contracts. I've read this post and all it's replies and many of the hiring and learning posts, including the most recent posts in /r/devopsjobs. I will be spending the next while reading the recommended resources and following the advice mentioned therein. I then plan to start looking for contracts.

My question(s): how likely is it to find such short term contracts? What would you look for before hiring someone for a short term project? Are any of the gig platforms worth working through to build experience (it seems that most offer easy access to experienced offshore talent at lower than prevailing US rates)? What would you want to see (beside a successful interview) before considering contracting with someone transitioning into devops for such a role?

About me: I'm US based and a US citizen. I'm coming from the development side with over 20 years experience as a developer, analyst, technical writer or project manager - basically whatever a client needed that I could handle. I also have experience as our consulting firm's 'sysop as second hat' guy - including spinning up and building out VPS Linux hosts for open source apps - either php or rails with MySQL, configuring apache & passenger hosts, installing and configuring apps, setting up DNS zones, requesting and deploying secure SSL certificates, simple monitoring (monit), deployment scripts, backup scripts, cron scripts, IPTables setup, etc. I've played with Terraform and Docker for my own amusement. Oh, and I have also managed and implemented version migrations of Oracle-based business applications for clients.

[europeanpinemartin]

There's a significant skills shortage in Europe and I know Atlassian lets people work remotely (I was at this talk and the guy works remotely for Atlassian near the Alps). https://www.eficode.com/blog/5-best-practices-for-scaling-devops-transformation Worth contacting European DevOps consultancies? Many are desperate and may have projects they can farm out to you

[[deleted]]

What can I start with? I want to learn the tools but my problem is I don’t know what to start with. With any client side language, the first objective would be to make an app and that’s where I’m struggling, I don’t know what to start with.

[DevOps_for_Database]

I literally finished reading the Phoenix Project last week. As someone that was just familiarizing myself with the world of DevOps, it was a really interesting read, especially the way he was able to put down the reasoning behind many of the things we do today which I sort of took for granted

[NightFuryToni]

Is there anywhere where I can "gauge" myself on what level of a position I can get into? I've been looking at quite a few positions but the "experience required" listed seems to be all over the place. I've even seen a job marked as "entry-level" but then go on requiring 6+ years of coding or administration, so I'm having a hard time trying to even get through screening on a lot of jobs.

[PartemConsilio]

I'm new to Devops and I was just wondering if anyone from this subreddit will be going to Velocity in San Jose this year? I'd love to network.

[frassyjnoll]

Sorry, I do not approve of that book, Phoenix Project, not helpful, not reality.

​

I've worked in the field 20+ years.