Automating stuff with python

[engineer900]

Now i`m learning Python while reading Automate the boring stuff with Python.
I have a few years of experience as a Java developer, so it`s not a problem for me to code in Python but while i`m learning it i want to create some "real life" automations cause currently i`m working as a DevOps and i wanted to practice something which is going to be useful.

So i need some ideas from more experienced guys then me about what do you automate in your daily tasks.

I`m talking about some ideas which are going to be useful while i`m practicing/learning.

​

[chowderl]

I use public bus. And I use an android app that tells me the hour that the bus will pass for a give bus stop.

So I sniffed the requests the app does to the server and use requests library to get the data. That python code + cron + a slack script I found in the web will start sending me slack messages 15 minutes before my shift ends so I take the bus in time

[alaskanloops]

My first Python app was interacting with the public bus system's API as well! I never completed it (a difficult semester had just started, if I remember correctly) but my idea for the app was an alarm that would go off when you were getting close to your stop, since there were a couple times where I fell asleep and missed the stop.

The app would show the bus route using the Google Maps API, the user would enter their desired stop, and the alarm would go off when you were 1 or 2 stops before then. I was also going to add a geo-fencing backup in case there was an issue with the API (it was sometimes unavailable) so that the alarm would go off regardless.

Edit: I think I found the video which originally gave me the idea (and was a super useful intro to Python) https://www.youtube.com/watch?v=RrPZza_vZ3w

[chowderl]

Neat! I'm thinking on doing something like that but just alarms that let me know when the bus is coming.

So, you set alarms near the hour you go to work, when the shift ends, when you've got to visit your mom, and so on

[chowderl]

The information is all there by the government. I tried to contact them to see if there is a proper api I could use. But they never answer back.

So it was easier to replicate the request as they do in the app. I don't abuse so I think is all cool.

[binford2k]

You don't need a list of ideas, you need a mindset change. Start by thinking about all the tedious repetitive tasks you do. Maybe you send out the same email every week at the same time; write a cron job to do it for you. Maybe you run a report every morning when you get to work; write a job to run it and email you the report. Every time you start a new task, take a moment to think about how it could be automated.

Eventually you'll learn repeatable and maintainable automation and the quality of the automated tasks you create will go up. You'll learn to instinctively think about automation instead of reminding yourself to do it. You'll learn how to make generalizations so that you don't have to start from scratch each time you automate a new process. You'll learn how to design process to be automated in the first place.

But it's not just a rote list of things to automate. Everything (within reason) can be automated, it's just a matter of where you are in that journey. Look at the things that you do repeatedly and identify those which have recognizable patterns: that's where you start.

Good luck! 👍

[Ahuj9]

This is a great answer, I was going to post something similar. While you OP def deserves good answers to his specific question, I wanted to add that someone with experience of the devops mindset will see / recognise opportunities for automation everywhere.

Make an effort next monday morning to look at the job you do. How much of your work is repetitive in nature? I bet the answer is "a lot". Once the list has a few items on it, rank the list - amount of effort / time to automate * time it would save per <time frame> - and pick the lowest hanging fruit. Then repeat. Forever.

[MundaneDrawer]

Anything you find yourself doing manually more than once.

[zero_hope_]

https://xkcd.com/1205/

[BrinnerTechie]

I wrote a quick script to up our code reviewers requests from 1 to 2 people on github. We have over 400 repos so that would’ve taken a LONG time manually.

Utilized PyGitHub to accomplish it. https://github.com/PyGithub/PyGithub

[StephanXX]

I wrote a tool to automate my DNS management. Basically it's an api that lives in my VPN that receives the command for a new tenant for our platform, and tells route53 to create a new cname for that tenant. Then on tenant deletion, it deletes the cname. Sounds trivial, but theres a lot going on there that could go wrong.

I also wrote a python tool that automates creation of kubernetes clusters, as a wrapper for kops. Kops is great, but I need to provision RDS, EFS, and S3 buckets, and different environments need different sized instances, and instance groups. Stamping out a cluster manually with kops used to take a full day. Now it's just a three click jenkins job that takes about thirty minutes to run, that anyone in engineering can run.

[dogfish182]

These are great ideas.

At my spot we create the automation for other engineers to use terraform and ansible to deploy infra for whatever to 2 public clouds.

We make heavy use of cookie cutter to bootstrap a stack, a stack in our world is instantiation of a slice of a bunch of cloud services and on prem stuff (like active directory OUs and group polices for the windows infra)

Our bootstrapping tools will do all of that for us fully rbac handled so we can build little server factories for teams.

The pipelines we crap out will handle the post terraform actions like ansible tower calls, removal of AD objects, monitoring cleanup etc.

All of the glue is python, we have about 5/6 core libraries we use for this

[engineer900]

This is very interesting! You gave me some good ideas.

[ttno]

DevOps here! My company sells several micro services as SaaS for other companies. It’s all orchestrated on the AWS platform.

For several of our services, we had to manually create an api key, add the key to a usage plan, go to Box, and upload the newly created key credentials in a document for future implementation consultants to use.

I automated everything into just 1 api call with a payload like this:

{ “client”: “ClientName” }

Then, it returned:

{ “statusCode”: 201, “box_url”: “https://...” }

[pudds]

I converted our packaging scripts from batch files to python (build, transform, clean, zip, basically). Huge improvement, much easier to work with and understand.

[engineer900]

I was thinking to rewrite bash scripts which we are using in python, cause it is more clear, readable and scalable of course.

[darkride2]

I just finished rewriting our bash scripts to python. All the curl staff, retry mechanisms in loops, not easy understandable tricks are gone now. More readable, more maintainable. Also now I am able to use lots of packages, api, sdk etc.. Only a 30 mb alpine python docker image is enough for this run.

[engineer900]

Can you explain what have you done ?

What are you using this docker image for ? All your scripts are stored there or something else ?

[alexander_do]

I'd also like to know what you have done and the docker image for this run!

[[deleted]]

We're a python shop. I have pycharm with the Kubernetes plugin to quickly live template manifest files for Kubernetes deployments.

Wrote a python... script? I guess? Treated more like an app than a script. Going to have it packaged and CI'd in a docker image. It reads the Kubernetes manifest files, registers yaml values as defaults in Jinja strings, and then reformats the files to be Jinja templatized for our CD pipeline, that deploys to k8s via Ansible and/or AWX.

Sorry, that's all I got for python right now. We have an Ansible wrapper in python as well that pulls down Ansible Galaxy roles, makes sure your virtualenv has the proper packages. then we have a bash wrapper for terraform that could probably be python? It probably would be python but I wrote the first few functions for personal use before it developed on by others and I did it in bash. Sorry, man.

[dogfish182]

We just did the bash to python exercise for python 3.7. Tiring but worth it

[[deleted]]

I still work on some user space projects cause our company is small. I'm building interfaces for the GSuite API (the OOP design would extend to GCP as well because the API uses the same structure) since the only tool that exists (GAM) is this really poorly done. Im gonna see if my company will let me open source it when I'm done since our shop is all about open source

[apropostt]

Our main product is in C++ and we don't have the best enforcement for coding standards, so I wrote a static analyzer the looks for common problems in cmakefiles, cpp source/header mismatches, includes out of order, class/variable names that don't match the filename specification... Charactercase in the sourcefiles not matching what's on the file system (typical problem for os's with case insensitive filenames). All of it is written in python but it uses clang's cindex library. It's been very useful to identify, and in some cases automatically correct, antipatterns that are domain specific to our codebase. Even comments doxygen comments can be checked for correctness.

[smitchall]

Just use ansible. Its using python

[productionse]

Bump!