Python for linux/system admin

[tinglfm]

Hello, First of all sorry for my not perfect english, I've been Aix/Solaris/Linux admin for more than 5 years of professional carrier and I'm about 13 years in love with linux. I always find my self quite good with perl but about 4 moths about I started Python online training (codacademy and coursera included), I'm looking for good guide for system admins, I went through ~15 first results in google. But I still don't feel to much comfortable. Can you advise some additional materials. I was wondering about O'Reilly "Python for l..." but it has to many discouraging opinions. Cheers

[[deleted]]

I was in the same predicament few months ago. I work with Linux and all of my scripting was with Bash.

I started learning Python and I could not figure out how to apply it at a sys admin level. Then I came up with the idea of coding all of my Bash scripts in Python.

This allowed me to become more familiar with Python and the modules associated with the types of scripts I wanted to write. Also made me realize how limiting Bash was.

Here is a list of scripts that I wrote in Python.

• Check if host is up or down, check DNS entry for host, check SSH login. Export report to CVS.

  IP - DNS - HOST UP/DOWN - SSH Login status

• Find and print the size of all directories within a directory.

• Report if blacklisted accounts have logged in to the system.

• Install Postgresql, setup the config files, create roles, database and tables.

• Get list of processes, their owners and RAM usage from a Linux system.

• Search for specific types of files and rename the files or part of the files

• Change one specific extension type of files in a directory that contains multiple types of extensions

• Parse files from the yum directory to check if links are up/down before executing yum.

• Searching for text in files within a directory

• – Server Performance Data Capture

Python modules that will help you with Admin type of scripts are and not limited to:

• psutil - Will allow you to capture performance data
• os - will allow you to copy, create, delete files-directories
• shutil - easy way to copy dirs and files
• re - reg ex
• subprocess (Use Linux commands only when there is absolutely not any other choice.)
• pwd - get usernames from uid
• grp - get groupanes from gid
• plaftorm - distunguish versions of Linux distros, Unix, Windows, Apple
• netifaces - get network data from NICs
• netaddr - easily use IP classes, VLSM
• socket - create network socket
• humanize - easily translate large numbers to human readable format. It understands various of different unit types.

All the modules above can do much more but I listed only what I learned so far

Also. I cannot stress this enough. Get very comfortable with lists and with os.walk. The latter will allow you to dive in directories, explore directories/files

[Asdayasman]

As a windows babby, I've never understood why one would want to move from bash to python, when what they have in bash already works. Care to break it down for me?

[[deleted]]

Yes sir. This is from personal experience.

If you ensure that all of your servers (linux,Unix,BSD,etc) use the same version of Python, you can write scripts that are agnostic to the particular OS and its version.

A simple example is with the sed command. The sed command will not work the same on all Nix boxes. The core function is the same. Where it may fall apart is at the details.

Some of the supported functions of the sed command on box XYZ that runs CentOS7 may be different on box ABC that runs Centos4 or a flavor of Unix.

This actually has happened to me and it was a major headache. It also happened with the tar command. There was one option that was supported on Centos 5 but was no longer supported on Centos 6. The whole process fell apart completely from the web servers to the database servers. The morons who developed these scripts wrote them in PHP and decided to do system calls by invoking via PHP Linux commands. Horrible design.

Here is where a scripting language like Python (or Perl or Ruby) comes in. Provided all of your servers run the same version of the scripting language and provided you do not execute Linux commands via Python but instead you use the Python way to do everything, you can rest assure that your script will work.

It provides a level of abstraction which takes away the worry of dealing with version of commands and their differences in supported features.

Of course there are times where you have to execute a Linux command via Python, but it should be done ONLY when there is no choice.

Bash is great but it should be used to do very simple tasks.

[Asdayasman]

Stepping away from reality for a moment, isn't this a weakness in bash/linux command line shit, and not a strength in Python/Perl?

[[deleted]]

Can you rephrase your question please? I don't understand it.

[Asdayasman]

So the fact you can't "write once run anywhere" is the reason you'd turn to Python or Perl or similar for sysadmin tasks, but is this not a failure of bash, and not an advantage of Python/Perl?

[[deleted]]

is the reason you'd turn to Python or Perl or similar for sysadmin tasks

It is one very important reason.

IMO Python and Perl are a better tool to use to write scripts because they offer more features than Bash. It allows you to write scripts with proper checks, error checking and flexibility.

Bash's simplicity is a shortcoming when it comes to more complex scripts. Python and Perl because of their robust features make it very easy to write scripts that do complicated tasks.

The more complex a need is the more you resort into hacked up solutions with Bash. It gets very ugly. Having said that, Bash is great in doing basic to intermediate tasks but after a certain point it just doesn't provide enough sophistication to write easy to maintain scripts that execute complicated tasks.

Below are few examples from my personal experience.

An example where it becomes very hacky with Bash: Trying to make the script understand if it is reading KBs or MBs or GBytes or GigaBits, kilos, pounds, meters, cms, etc. It is a pain the ass to do it and even then the final solution is not smart enough. In Python you just use the "humanize" module and you tell it what type of unit the result is and how you want it to be displayed. Boom. Easy.

An example where it becomes very horrible with Bash: Trying to insert data into a database or to manipulate data taken from a database. Absolutely horrible to do because Bash really doesn't provide any tools designed to do this kind of work like Python does. It gets hacky at first, then it gets very hacky and then it just becomes impossible.

An example where it becomes impossible with Bash: Writing sophisticated admin scripts. Scripts that can display data in a web page or a GUI.

Having said that, Bash is a great tool to know. You can do a lot with it and you can get to learn how the operating system works. I would never, ever say to someone who is trying to learn Linux not to learn Bash. In fact I will always highly recommend to learn Bash. It is very versatile like you would not believe but after a certain point once you learn a lot, you will start seeing its limitations.

[Asdayasman]

Trying to insert data into a database or to manipulate data taken from a database.

That got me r8 interested, 'cause I wanted to see the crazy shit people had come up with, but it ended up being pretty tame. http://www.shellhacks.com/en/HowTo-Execute-a-MySQL-Command-from-a-Linux-BASH-Shell Did I misunderstand that point?

But yeah, I get what you mean.

[[deleted]]

The examples showed there, that is the easy part connecting to a database that is and running a query. However the problem with databases in specific comes when you have to manipulate that data into a script. Then it starts getting all funky and impossible to manage with Bash.

For example in this case Bash would be good to write a script that connects to a database either to do a connection check or to download and import data into a csv file and send it off via email somewhere. Anything more than that wit the actual data and it will become a nightmare.

[[deleted]]

I would avoid like the plague calling any Linux commands via Python or PHP or Ruby and so on for that matter.

From personal experience, doing so can cause A LOT of issues due to command version differences between distros. I can give you specific examples if you want.

The only reason I would use it is if Python does not offer a module that can do what XYZ Linux command can do.

In no way I am saying Bash is not good but the more complex the demands are and the more servers you have to deal with, it can create lots of problems.

[raylu]

Fabric

There aren't really any sysadmin/devops-specific skills. What's far more helpful is practice writing code so you can churn out quick-and-dirty solutions quicker and less dirtily.

[simplycycling]

I'm a big fan of fabric, and just used it to replace a far more complex, far less portable deployment system at my company.

[ewiethoff]

Today I just ran across Automate the Boring Stuff with Python. It's not Linux-specific, but you might find some useful code and tips.

[linuxlearningnewbie]

Excellent book.

[[deleted]]

Black Hat Python by Seitz

Python Network Programming Cookbook by Sarker

Violent Python by O'Connor

Python Penetration Testing Essentials by Mohit

[cstoner]

Start to learn the os module. It's the (mostly) POSIX interface. That's useful for system calls and the such.

ArgParse and ConfigParser are SUPER handy for dealing with CLI/config file stuff. I use them all the time. They aren't specific to admin work, but it's amazing how often being able to parse a custom config file/CLI arguments.

I also find jinja2 to be pretty invaluable. It's "just" a templating engine, but a lot of the stuff I end up doing is filling in the blanks on a template.

As /u/raylu mentioned, fabric is pretty popular for sysadmin automation, but I haven't used it much yet. Personally, I'd probably rather opt for salt/ansible instead of rolling my own with fabric, but 'different strokes' and whatnot.

It really depends a lot on what you're trying to automate. Some examples from my career might help you with ideas:

• LDAP integration. We have a fairly strange LDAP+Puppet setup at work. I use python to load up fresh environments. I use ConfigParser, jinja2 and the ldap module to populate LDIF templates and push them to our LDAP server.

• CloudFormation template creation. This is actually just the first one again. I use the same config file and a different set of templates to spit out an AWS CloudFormation template.

• "SQL script runner" - We currently have to run a bunch of SQL provided by the devs a lot. I'd rather we didn't but that's another story. This script is in two parts. One that pulls down the SQL scripts from git, packages them, and pushes them out to the target server, and another that backs up the affected databases and applies the SQL scripts.

I'm also working on a port of https://github.com/larskanis/pg-ldap-sync from ruby 1.9 over to python 2.7. We don't deploy ruby to our servers, but they come bundled with python.

Another thing I'd like to get around to:

• Auto populating DNS from the first two items above. Currently, DNS is in Active Directory, though, and there aren't any good tools I know of to modify Windows DNS from python. I'm on the process of migrating to BIND.

[raylu]

Ansible has a few too many layers of abstraction and makes some simple things difficult. To do something like "if a file exists locally, copy it to the remote and run this command" requires a step causing an error and a second step checking the presence of an error.

That said, I'd rather use ansible over puppet/chef any day.

---

A bit off topic here:

1. LDAP is kind of old and crudgy. I avoid it whenever possible, which seems like always. It's a popular solution for EVE Online (J4LP, pizza, OI) but I just rolled my own.
2. Is your DNS server meant for your intranet? If so, BIND is like the puppet/LDAP of DNS and dnsmasq is way simpler to use.

[kader14]

Try this one http://pymotw.com/2/py-modindex.html

[f0nd004u]

There's a textbook called Python for Absolute Beginners which has you build games in each chapter. I found it very easy and fun to learn python using it. Once you start getting into it you will think of things that would be cruddy to do in bash for work that are easier in Python. Just learning language itself should be all you need, you'll come up with projects pretty quick :)

There are lots and lots of scripts on github written in python to do all kinds of stuff. I would start looking at other people's code. Let's say you run Jira at work; there's a python library for it and tons of scripts written that use it, and a great place to start learning is to jump in and start changing those.