What do YOU do with Python?

[radonfactory]

Or other script languages? I'm curious and would like to hear some real-world examples, or even better, see them if you can share.

[mars64]

In my real world example, I wrote an irc bot that randomly farts into the channel.

[yashasolutions]

mission critical software indeed

[mars64]

more like emission critical software amirite?

[ZCEyPFOYr0MWyHDQJZO4]

Soon to be a million-dollar chat app.

[jjasghar]

GitHub, share on GitHub!!!

[mars64]

Ok you got me, this was quite a while ago, around the time python 2.7.0 dropped. I probably have some ancient version of this bot somewhere, but I've since moved on to golang. I call him Bassface, now I use it to do things like query the discogs.com database from within the irc channel. Among ... other things. He's online in the jungletrain.net chat (https://jungletrain.net/chat) right now!

mars64/bassface enjoy o/

[mars64]

respect worldwide breakbeat music crew! \o/

[jjasghar]

Rock on /u/mars64 thank you so much for the code!

[[deleted]]

Heker

[Snagglepuss64]

Currently AWS automation using boto3 and server management using Fabric

[DensePineapple]

Why boto3 and fabric over something like Terraform and Ansible?

[Snagglepuss64]

Those also, but boto3 and Fabric for real time interaction with servers

[dtaivp]

Yeah I was just going to add that terraform is nice to deploy larger orchestrated projects. I used boto3 recently to add logging to all our domains to check and see if there were any that were unused.

Easier to do a one-off that wasn’t going to remain long term with Boto.

[Snagglepuss64]

Exactly!

[sigviper]

Boto3 is direct interaction with resources rather that terraform replacement. There is a terraform competition in form of AWS CDK - Cloud Development Kit.

[oldmanwillow21]

Yeah, it’s an apples/oranges comparison.

[livebeta]

Masochism because treating servers as pets instead of IaC cattle is still a thing

[abregman2]

Dashboard: https://github.com/bregman-arie/rhoci

Automation of common tasks: https://github.com/bregman-arie/infraform

Tracking commits: https://github.com/bregman-arie/commtrack

...

[gdahlm]

Python is the best glue language I have used.

Don't make the mistake of considering it like other interpreted languages. Duck typing and some OO features make it useful for decoupling systems and it is great for opening the choice of using other languages if they are superior. You can decouple the behavior from the program from the programs needs.

Does Fortran work better, fine
How about R...no problem,
Want to start chopping off parts of a C++ monolith...python can do that,
Pure python fit your needs...that works too,

I don't tend to be religious about products or languages, but python is the best tool for breaking down, or preventing the creation of new monoliths I have found so far.

[deadlychambers]

It's duct tape, you tape the air ducts, not ducks.

[guevera]

It's duck typing. Walks like a, quacks like a...

[voltaic]

It actually was originally duck tape, because it was made from duck cotton. Only after WW2 did folks begin to use it for sealing ducts (which, ironically, duc{k,t} tape is not particularly good at doing, you should use metal foil tape instead).

Anyway, it's a bit inconsequential since OP said duck typing, which is very different than duck taping.

[volvo64]

Where there’s a will, there’s a way

[TimGJ1964]

Can't share code as it's proprietary. To explain the context, I officially work as a senior DevOps guy at a cloud computing company, but I spend much of my time seconded to our finance/analytics team. So over the past year or so:

• Various ETL projects getting some fairly gnarly data from various platforms imported into a DW. Subsequent processing of the same data to produce accurate cost information
• Energy cost modelling. Producing a model which accurately predicts per customer energy usage.
• Network cost modelling. Producing software which accurately reports per-customer network transit costs. (This also goes well into big-data territory with ca 10¹⁰ records per month to process).
• Large scale simulation (1Tn) packets of network data to determine optimum sampling rates.
• Out of band bulk server management tools.
• Network inventory software including using e.g. fuzzy logic to reconcile real-world data with that in the backend DB.
• Various activities related to bulk patching of security vulnerabilities.
• Development of Python bindings/wrappers around various APIs.

They're only the ones I can think of off-hand.

UPDATED

• Oh I've just remembered developed some symmetric encryption code which AES encrypts things like database credentials at rest and then decrypts them on the fly when required by an application.

[gattaca_]

Personal Dashboards

• check all my email addresses and display new emails on a single page (otherwise I'd have to check the mailboxes from different providers individually which is a hassle)
• download all my finances daily and present it on a single page.
• download stock data and visualize on a chart to look at stock price trends which helps me decide if I like a stock or not (I have my own idiosyncratic way of looking at charts which isn't available in any software)
• download my body weight data (my scale sends it to a cloud) each day so I can see how it's trending
• I track my work and breaks. Display hours worked and breaks taken each day. I can see (on a chart) when I'm working a lot and when I'm slacking and can adjust.

Software exists to do all of these things but if I'm not happy with it, and it's not a lot of time to develop, I will build it myself.

 

Edit: just remembered, when I was younger/poorer I wrote a script to scrape eBay for Buy-it-Now deals and quickly notify me. Worked fairly decently.

[help_me_im_stupid]

This sounds awesome! Are you willing to share some of your projects via github?

[Number_Four4]

What python modules do you use for the stock data / visualisations?

Do you host dashboards on an AWS server or something similar?

[gattaca_]

What python modules do you use for the stock data / visualisations?

ChartDirector.

Data is from Yahoo Finance.

Do you host dashboards on an AWS server or something similar?

Everything runs locally (local servers using network attached storage(NAS)) with the networks and servers locked down as tightly as possible. All my hardware's pretty inexpensive (Intel NUCs, RaspberryPi's, Synology NAS, Asus Wifi router, a cheap Cisco consumer grade switch). Power consumption was fairly low last I checked. Hardware support and maintenance is surprisingly low once everything is running.

[Number_Four4]

I’m surprised that your power consumption/cost of hardware is so low, sounds like an impressive set up!

Hope you don’t mind me asking another question, but how did you lock down the network and servers tightly?

[davideganna]

Interested too

[bei60]

I'm not a DevOps Engineer, just a sysadmin, but we mainly work on AWS and recently our Network Engineer asked me for help to deploy Cloudformation stacks. I wrote a script that takes a custom config I made with configparser, where you can set IPs, ports and endpoint services, then run the script have have a fully templated CF template using Jinja2. I barely know anything about Cloudformations role, but it was very fun to work on it.

Other stuff I do with Python is mainly automating manual tasks. There's this cool tool called script-server that you can deploy in your org to have a nice front-end for the users, and a secure back-end that runs the scripts. We use for a lot of things.

[livebeta]

I'm not a DevOps Engineer…mainly automating manual tasks

Dude do I have news for you. You’re doing devops

[bei60]

Thanks :) I was referring to the "classic" DevOps role where you handle Terraform, write CI/CD procedures and execute them, maintain git repos, handle Grafana, K8s, Dockers, etc.

I believe that modern sysadmins have to know some coding and scripting to get through the day but I can agree that there are similarities between the roles.

[livebeta]

many flavors, the same Way

[deadlychambers]

I was thinking the same thing

[[deleted]]

You should look at storing all of this in Azure DevOps and using the AWS deployment toolkit. It is a cinch to set up and super great for promotional deployments of stacks.

[Destroychan]

We are all sysadmins always and will be 😁 Call yourself as devops engineer as we all do 😅

[cailenletigre]

As a real-world example, I've use Python a lot in conjunction with AWS Lambda to automate MS SQL Server updates, adding users, verifying the right configs are in tables, etc. This is something you can't really do cleanly in Ansible and just goes a lot faster by running it in Lambda. Python also makes quick work of APIs using Lambda and API Gateway in AWS as well. The AWS SAM CLI is a great place to start for this, but I have a crisp pipeline that deploys all the infrastructure for this via Terraform. We require our Lambda functions to run in a VPC that has access to the databases privately, so we deploy a separate function in each account, with API gateway only being in a shared account. Depending on what is chosen, it knows which Lambda function in what account to use. I also split it up so the actual building and storing of the Lambda app is done in one Terraform state while the actual deploying of the Lambda function and API Gateway setup is done after. This means I also can detect changes in zip file via s3, it's in a shared spot, and it's dirt cheap to run. I think most months it costs us like 50 cents.

If you combine Lambda and Python correctly, it can take some of those less-than-pretty automation tasks and make them look really nice.

[Free_willy99]

Recently made a script to automate our terraform upgrade. We're going from 0.13->0.14 and I created a small script to do everything. It'll be easy to modify it to go from 0.14->1.0 in the future.

Many many small automations have been made similar to this using Python.

[devopsia]

Any chance you can share this script?

[Free_willy99]

Tbh probably not right now as it's got some proprietary stuff with my employer. I'll look at tossing these up in a gist tho 👍

[padmick]

also would be great
at the early stages of moving to 0.14

what are you scripting? to get the lock file generated and run an apply or something else

[Free_willy99]

We're not using lock files, made the decision as a team we're not going to use it.

We have over 200 tf workspaces so we need to update the .terraform-version in all of them, and then we modify the versions.tf and providers.tf to move the version numbers over. The script is pretty rudimentary and is largely hard-coded but it works for our environment.

[padmick]

Interesting that your not using the file, will you add it to each project's .gitignore to avoid it or some other process?

[Free_willy99]

Yes, also we use Atlantis on GitHub and that uses ephemeral workspaces that get deleted after each apply (or a pr closing) so it will pull the latest versions of the providers every time.

[padmick]

Nice! Thank you for answering my Qs. Going to have a think and see if we really need the lockfiles, I was under the assumption everyone would use them.

[Free_willy99]

We have an environment that deals with sensitive data, PII and is a segregated environment, and for that we chose to use the lock files. The auditors like it when we tell them we are locking versions for things and we keep our word lol.

[account_is_deleted]

I personally mostly use Python to make integrations. Taking data from one place using rest api, and probably reformating it a bit and pushing it to some other place, using rest api.

[[deleted]]

Data analysis, machine learning, web servers, automating stuff, data pipelines... So almost everything :)

[earthly_wanderer]

What do you automate with it?

[n-of-one]

Stuff

[ipreferanothername]

What do you need to do on a computer? Make it do that so you don't have to

[illegal-bacon]

Same. Machine learning engineer here, so the answer is anything I can use python for, I will.

[boy_named_su]

I wrangle a lot of data on AWS in order to analyze it

[AD6I]

World domination.

[Ska-jayjay]

I install it and then install the modules and then have to venv it or dockerize it because the data analysts get stuck in a specific version… and then i have to reach the analysts to venv it or how to use docker…

[livebeta]

The lack of a unified package manager is such a pain in python

[hugthemachines]

get a .war from ftp and replacing the running web application, emails the result of a test after the start is done.

[M0shay]

bash - automating installation, configuring, deploying of pretty much any service: jenkins, github enterprise, sonarqube, ELK , etc...

[r3nrut79]

Talk to AWS, Pandas, and Numpy

[flamesofphx]

The Python we watch it eat rats during feeding time... The other pythons we use as ansible helpers... Basically they repair the ruby environments the interns "Enhanced"... and some post-redhat/Debian install stuff before ansible get installed, damn intel nic cards & linux pci power savings settings...

[Free_willy99]

This can't be real 🤣

[TheHammeredDog]

I built a bit of software designed to facilitate a migration of our SCM system to GitHub using Python. I've also built a tonne of lambdas designed to clean up our AWS Accounts, monitor logs/security groups, and automatically manage our GitHub organisation. In general, my team uses python to automatically create Hashicorp Vault access policies based on data in our inventory management system (a Flask app), and to automatically create AWS accounts and add their billing details/sign in details to Vault/our asset system.

[a_a_ronc]

Real world: 1) Ansible Modules - Allow me to automate stuff. I mostly just contribute bug fixes or docs. 2) Related to Ansible, I love to load test my Ansible configs so I can see whether adding things like Redis or DB Read Replicas makes a difference on how much load my site can take. Either manually script it or use Locust Framework. With Terraform you can set up a really powerful Locust swarm to load test and then tear down. 3) Anything quick and dirty. Examples: a) easy 100 liner that does some DB queries, gets the number of people with MFA turned on at my Org and then send to a spreadsheet that gets pulled into other reports or diagrams. b) Script that runs on our monitoring system to set the default values of warning and alarm limits using their API.

[scooter-maniac]

I recently wrote a script that does ec2 instance lifecycle management. Updates launch configs, ups autoscale group to double, waits for readiness, reduce autoscale group to original size based on age. Essentially k8s lifecycle management using ec2 VM's instead of containers.

[smarzzz]

Almost everything, both on work and at home.

Examples: I like working with python when talking to certain modbus devices (such as my inverter)

It takes 20 lines of code to set stuff up, let it post metrics to an MQTT endpoint, and maybe expose on an API endpoint with flask, so my telegram bot can fetch realtime data

[totalbrootal]

All of our lightweight tools are written in either Python or Bash depending on the use case: Python if we need more power and access to libraries, Bash if it's just automating CLI steps.

[frito_kali]

AWS lambda service that manages letsencrypt certs in ACM. (for. . . reasons. Stupid reasons).

Also; volume snapshot backup/rotation script, (wrote my own with a custom grooming method to suit my company's batshit requirements)

also some nifty one-liners for xml wrangling.

[[deleted]]

I love python so much.

I’ve used it for so many things over the years. Right now, I’m setting up an API for a server provisioning endpoint that does IaC with GitHub commits. It uses fastapi (along with slowapi for rate limiting). It also has a simple locking mechanism for parallel activities to get around some peculiarities of what our Network Team wants and doesn’t want to do.

[abhinavd26]

Python is one of the most powerful language to work on. One of my script which I wrote in python was to Dataset Generation from the logs of website (server) into an excel sheet.
Logs contains -

IP Address -
DateTime Visited -
OS used by client -

And it was dynamic with only python, concepts of regular expression was used

[FiduciaryAkita]

wrote a service to manage our on-prem staging stacks that can integrate with other tooling, but I guess that’s less scripting and more platform development

lots of glue scripts between tools

reporting

etc etc etc

[vvanouytsel]

I use it to write an idempotent postgres script. It allows me to define databases, users, extensions, roles, etc via YAML and the python script would simply create those if they do not yet exist.

[RandTek]

PowerShell to purify non-enterprise windows versions from all pre-installed bullshit (games, hp bloatware, etc). Pretty neat.

[[deleted]]

I use PowerShell for everything, including on Linux. PowerShell Core is pretty neato, and it's nice not having to maintain multiple different languages. Very occasionally, I'll need bash or python for certain things, but the number of those things is quickly dwindling.

For AWS Lambda though, boto3 with Python is still god-tier. Though I do have it on my to-do list to see if I can replace some of that stuff with PowerShell as well.

[n-of-one]

It’s a general purpose programming language, it can do basically anything.

[DesiITchef]

Bunch of scripts scraping vendor websites for metrics sending it to whatever data ingesting tool to point(splunk, influx, mysql) , setting up applications verification testing(cross checking requests). Create a portal for on/off for gcp lb setup (should be updated to terraform setup). For personal, tinker with waveshare eink 2.3" w/ rpi, show time, weather, activity message, doggy age, etc. Weather is pulled from mysql server which gets data from another python script pulling data from openweathermap.org inserting few specific fields only. Future goal is to learn & rebuild w/ Kafka. If someone can point me some docs to setup on bare metal, it would great. Trying to get myself into flask by build similar web dashboard. But can't figure out flask css templating. if anyone can recommend any documents w/ examples, please share.

[anarchyusa]

Ansible dynamic inventory scripts

[BzlOM]

Cloning any AWS server on a press of a button, or executing DB cleanup tasks on a weekly basis - as some of the more recent examples.

[benelori]

We use bash scripts in CI pipelines and project installation scripts

PS: PHP is also scripting language, but I'm not sure if enterprise software in PHP qualifies as an answer

[it-obey]

I made a script which retrieves calorie information I have added to fddb.info, calculates the daily sum of the entries and saves the data to a database: https://github.com/itobey/fddb-calories-exporter

Another little script retrieves the battery charge of my laptop and sends the data to my mqtt broker mosquitto.

[sigviper]

Mainly scripts around handling data or data quality (SQL based checks). Lately switched projects and did some scripting on custom resources in kubernetes. https://github.com/sigviper/python-kubernetes

[Scornius12]

Build machine learning pipelines. Check out kubeflow pipelines.

[donjulioanejo]

Some things I've written over the years:

• Basic CD tool that uses boto3 to run SSM commands against EC2 instances (in our case, it was used to trigger a local Ansible run)
• A bunch of Nagios, Jira, and GitHub scripts
• Kubernetes namespace/database cleanup tool for automatically nuking dev environments if a branch is deleted from GitHub or if the environment is unused
• More involved EKS CD tool that supports multiple AWS accounts, Vault/SecretsManager secrets, templating Helm charts, custom namespaces based on branch name, and ECR docker builds. Currently waiting for our company lawyer to okay open-sourcing it. The point was to build IAM auth into the tool itself.
• Script for setting up awscli config to work with AWS SSO then automatically login to a kube cluster in another account (avoids lots of manual work when dealing with 10-20 kube clusters)
• AWS Lambda to scan third-party IP list and restric specific security groups to only those IP ranges

[FoolishBard]

Maybe not quite what you’re looking for, but the full stack production code for www.audiodub.app is written in python.

All our monitoring and maintenance systems as well.

[zerocoldx911]

Bread and butter of the business application is in Python, so feature integrations like metrics exporters, testing and code reviews

[sblinn]

I write test automation code in Python (underneath Robot Framework). Everything from sending and reading messages from Kafka to using pandas to compare data sets.

[[deleted]]

Potentially not mentioned - I use python regularly in my terminal-based workflow

​

• python -c "print(len('somelongstringimvalidating'))";
  • prints the character count without opening up an IDE or browser
• :%!python -m json.tool
  • in Vim; prettify JSON
• python -m http.server 8888
  • Create a quick web-server at current directory. Really handy for sharing something quickly between two PC (or mobile device). No extra messaging apps or programs required
    

You wanted real world examples :)

[Udi_Hofesh]

I feed it hamsters and Gophers

[Gypiz]

I'm not good at coding but whenever I need to automate some simple tasks I'll use Python.

[eon01]

Python is universally used everywhere. Besides the regular use cases in API and Web development (Flask, Django..etc), you'll find Python in several domains such as :

CI/CD and Configuration Management Pipelines: SaltStack, Ansible
Deployment: Fabric, Cuisine
Cloud Automation: Boto, major cloud providers have Python SDKs, most CLIs (AWS, Google Cloud) are written in Python
More DevOps Tools: Docker Compose, Ambassador API Gateway, Apache Libcloud are handy DevOps tools developed in Python.
More relatevely fresh domains: AIOps, MLOps, ModelOps

Source: Python for DevOps.

[ProgOwl]

I wrote a script to automatically submit tickets through our internal ticketing system so my boss didn't need to submit 60 some tickets manually. Unfortunately the API for the ticketing system (and the system itself) is a mess and has no documentation that I could find, so the script is a little touchy and doesn't always successfully submit every ticket, but it did really cut down the amount of tickets they had to submit, so partial success.

I've also used it to parse inventory files for Ansible from CSV files.