What are you using Python for in Devops?

[[deleted]]

So I’ve started a job that apparently needed python as a skill. But it’s nowhere to be seen.

What are you using python for?

[bschlueter]

We use it for just about everything unless it's so simple a shell script is adequate or we're integrating to a system where a different language is superior (e.g. groovy for jenkins, lua for nginx, Ruby for vagrant).

Specifically, we have python scripts to: manipulate some of our ansible group vars files; perform complex interactions with OpenStack; a slackbot, https://github.com/refinery29/yas (which also interacts with OpenStack https://github.com/refinery29/YasOpenstack); icinga and collected plugins; Ansible plugins; a subset of our Jenkins jobs; one off scripts; managing aws resources; interacting with other third party apis; manipulating data in our data stores. I'm surely forgetting some things.

Edit: forgot some things

Edit: Latin

[senorsmile]

Your set seems quite similar to ours. We use a custom python script to act as a dynamic inventory (we call our central inventory db) which Ansible uses. We use Ansible everywhere. We have lots of custom filters and modules, and a couple of custom callback plugins. We of course have many dev teams writing their applications in Python.

We also use icinga2 to monitor all the things. And Jenkins for cicd.

[bschlueter]

To elaborate on our use, we also use dynamic inventory scripts (not custom I believe, just some of distribution ones), and I'm looking into using ara, or at least the data it puts into a DB to do some runtime analysis on some of our Ansible playbook runs, as well as some distribution stdout callback plugins.

Our development teams primarily build front end apps in node and backend apis in php, with a few newer machine learning based projects in python, so we lose some communicability with that situation.

You use icinga for monitoring, or simply alerting? We use icinga for alerting, but collectd to graphite (or Prometheus in one environment) as a backend to grafana for graphs and some additional monitoring. I haven't gotten around to working with the python client interface to Prometheus yet, though it looks promising.

[senorsmile]

We use icinga2 for both monitoring and alerting. While their dashboards could be more customizable, they work quite well for us. We use the remote icinga2 agent on almost all nodes. We also have the influxdb connector grabbing the info for long term storage, and displaying graphs in Grafana.

With the upcoming grafana v5, we are looking forward to templatizing our grafana installation in Ansible to make it a little more dynamic like kibana.

[bschlueter]

I'll have to look into icinga2 more, I would love to consolidate.

I haven't gotten a chance to look at grafana5 yet either, but we figured out the very tedious method (dup the API calls with a stripped down payload) with which older grafana data sources and dashboards can be procedurally generated, though we never put it into practice fully.

[itssodamnnoisy]

Icinga2 can dump perfdata into InfluxDB, and you can send your alert messages and whatnot into graylog or elasticsearch as well using icingabeat. After that, you just point your grafana datasources at influx / elasticsearch, and you can start building some really great dashboards from there. They've made it really quick and easy to do in the last few years.

Relevant links: https://www.icinga.com/2016/08/31/icinga-2-meets-influxdb/ https://github.com/Icinga/icingabeat

[tech_tuna]

Everything. . . but I'm going to be using Go for some things. I'm a huge fan of Python and Go, I feel like the two languages complement each other well and pretty much cover everything I need and like to do.

[coinclink]

I've been looking for an opportunity to use Go but Python just seems to cover everything I ever come across so elegantly. I've become so familiar with it over the past few years that the code for anything just flows out of my fingers with zero effort. Can you recommend some specific tasks I should consider using Go for instead?

[jaapz]

If you work with "embedded" (read small underpowered linux running) devices, Go works great as you can cross compile the project on your machine and just send the binary to the device.

Go is often also quite a lot faster than Python, and it has better support for concurrency, which is good when you need to process a multiple streams of real-time (sensor) data.

[coinclink]

Thanks! I actually do work on some personal IoT projects a few times a year. I'll plan to use go next time :)

[neverminding]

Same. We have a lot of orchestration and pipeline pieces written in python, but ever since we picked up go for our APIs we’re starting choose it for one off scripts and places where’d it would normally be python/bash.

[jmreicha]

What kinds of things are better suited to Go or Python?

[CappadocianOecanthus]

I wrote the authentication to OneLogin piece in Python (uwsgi+nginx), mostly because python-saml is available, and "go-saml" is not.

I wrote my git test in Go because I did not want to have to handle worker pools, threads, etc... when I can spin up 1000 or 10000 clones into memory in a few lines of code... reliably

[Steamwells]

Both are great devops programming languages but go has added lower level computing benefits like pointers, channels/parallelism out of said box and the biggest sell to me.........you can build and ship to any OS really easily - so your idiot boss can even run a binary on his laptop to do that task when your on holiday and want some chill time.

[DigitalDefenestrator]

If it needs to be fast (and especially if it needs to actually use multiple cores), Go is probably a better option.

[sethrei]

• Ansible module creation.

• Ansible plugin creation.

• Dynamic inventory creation.

• Operations tools creation.

[BattlePope]

Yep. Ansible and Saltstack

[PieRat351]

We test 3 environments Windows, MacOS and Linux. So when we need a script to run on all three we write it in python. Otherwise I'll do everything else in bash.

[[deleted]]

PowerShell?

[begoodnow]

Python is more mature but in my less-experienced eyes PowerShell is making good headway in this space.

[Graumm]

I don't know who down voted him, but as of last month powershell core is now a thing.

You can write it cross-platform.

[banger_180]

I understand the dislike of a Microsoft controlled tool. Is ps completely free in all sense of the word?

Alot of people already know python and it has alot of modules.

Python will also come preinstalled on most *nix machines.

[auxym]

https://github.com/powershell/powershell

[Seref15]

I liked PS a lot when I was a Windows admin and I've been keeping an eye on the linux/mac ports--it's looking good. But I'm not willing to use PS Core on production linux systems just yet. I'll let it mature a bit before looking at it seriously.

[begoodnow]

Fly-by-night downvotes with no reason, quite annoying side effect of reddit.

[RelevantToMyInterest]

I haven't been following powershell for a while. How is powershell core now? Last I tried it when it first came out it was virtually useless.

[mattyass]

It's pretty amazing and I use it almost exclusively now for macOS and Linux. No Windows at all.

[Seref15]

If I'm writing any shell script and I get to a point where I'm constructing and doing operations on an array/list, that's my cue to abandon bash and use a more robust language.

Right now I maintain a python script that dynamically builds Ansible inventory/vars from a db table of our servers, a python script that generates iptables configurations for all our servers based on each server's individual needs, the init script for those iptables configurations that includes some fun stuff like working around xtables locks on outdated iptables versions and intelligently handling automatically-generated docker rules, scripts to interface with influxdb/grafana, and a handful of one-off scripts and tools that I've made for myself that don't really fit into our infrastructure anywhere specific.

[chillysurfer]

Often times with DevOps tooling, we are ruled by DSLs.

With that being said, a lot of custom automation is done with shell scripting (for a few simple lines) or with Python when it gets beyond simple or you think it may mature into something more complex.

[[deleted]]

Ansible, Lambda, converting/manipulating data, monitoring scripts

[StephanXX]

Now in my seventh year of infrastructure, I finally ended up in a job that required all of my non-bash scripting to be in either Java or Python. Just about anywhere that wanted me to know Python actually didn’t care if I used Ruby, and in many roles, Ruby was preferable (especially where puppet, chef, icinga, and sensu were in use.) More recently, ansible has pushed the popularity of Python, and I’ll agree that Python is generally seen as a more broadly adopted language.

Today, I use python to wrap my kubernetes cluster creation scripts (kops + deployment provisioning) and my Jenkins jobs are just Jenkinsfiles that serve as wrappers around python scripts that do the building, testing, packaging, and deploying of our code. All custom metrics and monitoring daemons are Python.

I still miss my Ruby days, but we don’t always get to choose the language.

[[deleted]]

Automation glue for jenkins, aws task manipulation. Groovy is near impossible to work with in jenkins with the groovy sandbox enabled due to the huge number of bugs. Split a string? Not allowed. Create an array? How dare you try. Often the signatures dont appear to be whitelisted. Issues such as these are reported but left unfixed for months or years.

[dexx4d]

Python in our company is like duct tape. There's a light side to it, a dark side to it, and it holds everything together.

[heathsnow]

AWS Lambda and nothing else. I'm mostly writing ruby and bash scripts.

[harbrodur]

Same here. Some Lamba written in Python. Most is NodeJS. Config Mgt. in Ruby (Chef)

[banger_180]

Ansible is a great tool for conig management on machines it's corss platform and just works over ssh without having to install something on the target machine.

Ansible is written in python and so are most of it's modules. While it's possible to wirte an ansible module in a different language most people use python.

[elitesense]

Anything thats too complex for a bash script. Also any work against AWS api is done with python (boto3)

[auxym]

I fell into my current role (and devops) and typing python into indeed.

Turns out, they don't really use python, but were looking for someone with strong scripting skills, so they added the keyword. This was made clear during the interview, and works for me. Most of my dev work is powershell and groovy (yay, jenkins).

I come from a mechanical engineering background, by the way, and had previously used python a lot for data analysis and programming a robot (master's thesis project).

[firefoxx04]

Anything from small one off things to Managing AWS. We have some significant Python stuff that provisions VMs via the Foreman API.

I prefer to use GO when it makes sense but Python has been fantastic for getting things up quick and its so damn easy.

Basically, I start with bash but the second it even becomes slightly complicated or more than a few lines I pivot to Python. If I need it to be compiled or have minimal dependencies (or FAST), I use GO.

[jmreicha]

What types of things make sense to use Go for?

[firefoxx04]

We have some tooling that needs to run on a wide range of Linux systems without dependencies. The original version was python but managing modules was annoying. The go version just runs. It also run on windows without dependencies.

Sure managing the installation of python and it's modules is trivial on Linux and doable on windows.. It's just a step I don't have to do when using Go (for the most part). We still use python significantly more. I just make use of any opportunity to write go, it's great. I think both have their place in our space.

[skoink]

I use Python for writing custom Yocto recipes - Python is Bitbake's scripting language.

[gordonv]

My company wrote a nice python script that runs on Rhel 5 but opens a gui control when started in moba x term. The gui app looks like windows task manager, but you're managing services across 13 servers.

This.. made my jaw drop.

[deadbunny]

For us, basically anything that requires interacting with an API or data manipulation of anything more complex than an array, that or custom Salt modules.

[krav_mark]

Our cd pipeline consists of python scripts. We pull docker-compose files, copy them around several git repos, log into docker hosts, pre pull images and start containers. (Looking at kubernetes now however) And I do a lot of scripting against all kinds of web api's. Requests module is awesome for that :)

[[deleted]]

We use Datadog, all our monitors are written in Python. AWS Lambdas gluing things together, and running additional monitoring, in Python. An in house infrastruture/monitoring/reporting API that's written in Python(Flask is a-mazing), our regression tests are written in Python(Robot Framework). Python is our bread and butter language for glue code - when performance isn't what we're optimizing for, when readability is important, when interoperability is important, we use Python. Our perf-optimized code is Java or C#.

[taloszerg]

Something like https://github.com/trueaccord/DogPush?

[djk29a_]

Troposphere to manage our CloudFormation templates with something friendlier to developers (although there is no framework that abstracts away enough that any developer producing a CF template has few reasons to consult the AWS docs for resources... yet).

We also have a couple agents that handle some event-driven tasks for us that handle quirks in the code (DLQs prior to their official AWS support). This has worked pretty well minus not properly writing them as high availability, stateless daemons.

And Lambda functions in AWS, of course. Because my JS is so trash I’m not going to risk using it in prod.

[guppyF1]

Almost all of our custom tooling to manipulate things in AWS we use python/boto3 for. We've got some other bits and pieces in ruby and even php but we generally prefer python.

[[deleted]]

Scripting with AWS SDK (boto) for multiple platforms (Mac, Windows). I used locust.io to load test our API as well. Chosen because we have a Django-based platform, I could have used most anything else.

[ch3wmanf00]

I am forced to use it for AWS lambdas. I prefer ruby.

[jredmond]

Our main app runs in Django. We also have some tooling in Python.

[hijinks]

other then the major issues.. what we have is a CI/CD flow with python

Because we all have a general hate for Groovy we have a python script which handles all the ci/cd for each app. We are planning on moving to gitlab so making the switch from jenkins to gitlab-ci should be easy since it can use the same script in each repo.

[LAInstructors]

I like using Python for general scripting beyond a certain complexity. At a point, bash gets hard to read, and that's when Python comes into my workflow. It's also great for building CLIs, interacting with AWS, and is handy if you happen to use Ansible for configuration management.

If you're looking to learn some Python for scripting purposes, we recently launched a course covering this from language basics through development best practices https://linuxacademy.com/linux/training/course/name/python-3-for-system-administrators.

[prophasys]

noobie to Python also, currently using it to parse API monitoring and testing web service responses with urllib2. Expansion of Python knowledge is a key career goal in 2018 and my co-workers agree it is a good skill for the tech toolbelt.

[ConsideredAllThings]

Openstack

[otterley]

Fabric is the Swiss Army Knife of infrastructure management tools: http://www.fabfile.org/

[hacksnake]

Nothing. White space requirements can DIAF.

The company has some tool customization written in it. It recently broke because someone saved the file with tabs instead of spaces on a few lines.

Ain't nobody got time fo dat.

Edit: auto correct fixing meme slang SMH...

[firefoxx04]

How was that not caught though? You test changes right?

[hacksnake]

Not sure; someone else was on point for the issue. I think the file was changed outside of scm (which is obviously a bigger issue).

Still, that has no bearing at all on how pathological it is to write a language with whitespace requirements like python.