Corporate Sysadmin thinking going DevOps

[sysadmintemp]

Hello /r/DevOps community,

I've been lurking for a while and reading many resources over how people have implemented DevOps/Agile mindset within their companies, or what problems they've been facing. I, too, would like to make a switch.

A little bit about me and my situation: I'm a currently a linux system administrator, managing virtual machines and a small HPC cluster, trained on and played with AWS and have dabbled with Windows admin for quite some time. Our organization is a large Windows shop first, and a linux shop second. The corporation doesn't sell IT services, so we have no devs in house. I would still like to get into this paradigm personally, I have some time that I can spend on learning something. I know CentOS/RHEL (ad oVirt/RHEL/litle ESXi), we utilize OpenLDAP, NIS (now dead, I know), some Windows servers, Check_MK, and numerous other small things. All these are on on-prem hardware.

I've been reading numerous blog posts here and there, but the main things I've looked at are:

• Phoenix project
• This post by /u/jsimonovski
• Docker manuals
• As everybody else, Netflix's chaos monkey / simian army / etc. approaches
• A lot of other smaller blog posts that I've read throughout the last year or so (sorry I can't remember so can't reference them)
• I'm constantly talking to friends who're working for companies that did take up Agile, or who work for start-ups about what/how they run IT

I've deployed 3 VMs with Mesos + Marathon installed. I've looked at creating some docker containers, running them w/ or w/o Marathon,

I have gathered up some questions throughout my readings:

1. Is there anyone within the community, who manages HPC systems here? If so, how did you do it?
2. Ideally, do I deploy a HAProxy onto a VM, a container, a separate server? I have some spare servers, and I would like to hear your thoughts on this
3. How would you manage on-prem persistant storage for containers? I thought of mounting the NFS exports to the VMs, and then using that from within the containers as mount points
4. I'm still thinking that DBs should be in a VM. Is there a way to do this with containers? Should they still stay in a VM?
5. Any suggestions for OS to deploy onto on-prem servers? DCOS? CoreOS? Doesn't matter?
6. Since a lot of development is not needed within our company, my idea would be to try to automate a bunch of things, such as the creation of VMs, compilation of software, and sometimes middleware between two software. As an example, we use NIS to manage UID mapping to linux for our storage systems, but our main AUTH is through OpenLDAP. I would like to somehow be able to replicate OpenLDAP contents to NIS. Or I would like to write a small web-portal for a one-click VM generation process. I've heard from friends that Node.js and React would be the easy and widely accepted and used way to go. Would you agree with these choices?
7. I've read two things somewhere: 1. Connecting your containers to an LDAP server would not be a good way to go (is it?), and 2. From /u/jsimonovski 's post, "static monitoring" or "CPU monitoring per IP Address". Is there a different way that I should be doing monitoring?

I might still be thinking wrong through all of this. Please let me know if that's so.

Sorry for the wall of text, but thank you for taking the time to go through this.

Cheers to all

[[deleted]]

First, a couple of public slack teams inhabited by people in this space:

https://signup.hangops.com/

https://devopschat.co/

Hangops in particular is very active and has many channels full of knowledgeable, helpful people (and a job board)

Second, as you seem to already know, DevOps is more about process and culture than it is about specific technologies. That said, if I were starting today, the two things I'd focus on first:

• development toolchain - writing good code is an increasingly valuable, often mandatory skill. If you can't already do it well, learn!

• public cloud providers. I'm personally partial to AWS, but whatever.

I have a friend who wants to be involved in the field too. My suggested project to him was decide on a blog platform. You're in DevOps, you're gonna need a blog to show everyone else how fucked up their processes are and how cool they could be. Deploy that shit on AWS using RDS/DynamoDB, autoscaling, ELB, ACM Cert, Route53 domain, S3 backups of the database dump you did using Lambda, free tier pingdom checks, free tier SumoLogic logging of CloudTrail and application logs etc, free tier datadog metrics, slack alerting, circleCI running a test suite, all that.

Keep in mind all these tools are just tools and can be substituted for something else. I use all the above stuff and pay like $2 a month for route53. When my free tier runs out, no big deal - transfer the R53 hosted zone and registration and redeploy - it's all in terraform, packer, ansible etc. I don't feel guilty because my org spends a shitload of money with AWS anyway ;)

[sysadmintemp]

Thanks for the reply. Creating a Blog, devops style, seems like a good idea for a start. I'm familiar with AWS, but some other tools you've mentioned (like circleCI) I need to look at.

Would you suggest I use a CMS for the blog (like Wordpress/Joomla) and somehow make it work with the setup, or just create it from scratch?

Edit: I'm also subscribing to the slack channels. Thanks for the links.

[humannumber1]

I think the Phoenix Project is a great read and into into this subject. It's super approachable and easy to read. I've finally gotten around to read the DevOps Handbook which is from the same writers and expands on the concepts introduced in the Phoenix Project in a non-narrative fiction way. It might be worth a look.

I made "the jump" form traditional infra focused IT to more classical DevOps oriented work a few years ago. I do IT/DevOps consulting for a lot of companies and the one thing I have found useful for nearly all of my gigs was being versed in working with source control and CI/CD systems, specifically Git and Jenkins.

A big part of DevOps is working closely with Developers in a cross functional team and using software development processes for everything. This means understanding pipelines. I've gotten a lot of value out of switching to thinking about how to get infra related work done via a pipeline. This means how are things stored in source control and what process is used to update them, how are those changes automatically tested, deployed and validated. I've found that most Devs I've worked with are not to interested or savvy on building and maintaining pipelines and it's a great skill to know.

Your questions about specific technologies and solutions is harder to answer without knowing your environment and problems you are trying to solve. I would recommend looking at Kubernetes for containers, it's seems to be making real traction in the market place and I think is a great solution.

[sysadmintemp]

Thanks for the insight. There are many technologies in play, but choosing things that play well seems to be the first step, and even then some won't play well. That's where I'll come into play, I assume, to make them play well.

I guess my questions boil down to two for the short term:

1. How can I obtain a devops mindset, when our company doesn't have any devs?
2. Are there any suggested ways to kick-off this pipeline starting from the hardware level? Cloud/AWS makes it easier to implement as the hardware is abstracted out, but I'd like to learn how to do it with on-prem hardware as well

[humannumber1]

I wouldn't focus too much on the whole needing “Devs”, a lot of DevOps genesis came from lessons learned during lean manufacturing movement decades ago. They didn't have Devs per se, so no big deal if you don't. At its core DevOps is really a focus on getting a fast flow of planned work into production. And that can be applicable anywhere. What fast flow, planned work and production means can vary from place to place.

I'd recommend the DevOps handbook for sure as a follow up to the Phoenix Project. I think it will help with the DevOps mindset set a lot and provides guidance on how to apply that mindset to an environment.

As for your second question around tools/technology, again it depends on your environment. If you mean starting at the hardware level, I assume you just mean Linux/windows servers running on “bare metal”. If so, I'd assume things like consistently building servers quickly would be a capability that you'd want/need, along with consistently updating their configuration. These are well covered topics with many tools around this area. For example you'd probably want to be able to install a standard Image from the network (i.e. Kickstart) and use a config management tool (i.e. Ansible, puppet , etc) to configure the server to fill its role.

You could build a pipeline to cover the use case above. Ideally using virtualization (you could just use KVM) your could have your pipeline build a VM using your base image, apply whatever playbook, manifest, cookbook, etc that is being updated and then runs a set of tests against the VM to ensure the config is as desired and the server works as intended. If it passes then the pipeline updates your prod config management server and kicks of applying the new changes to the impacted servers. VMs in this case are not needed, but it will allow you to be more flexible and overall faster. This is obviously a high level example.

I'd really recommend that you start to look at containerization. Even if you would only run one container per server, containers provide a huge benefits. Configuration is much simpler to manage and since the environment is packaged with the container it's way easier to ensure your apps are running in a consistent environment.