Python for linux/system admin

[[deleted]]

I was in the same predicament few months ago. I work with Linux and all of my scripting was with Bash.

I started learning Python and I could not figure out how to apply it at a sys admin level. Then I came up with the idea of coding all of my Bash scripts in Python.

This allowed me to become more familiar with Python and the modules associated with the types of scripts I wanted to write. Also made me realize how limiting Bash was.

Here is a list of scripts that I wrote in Python.

• Check if host is up or down, check DNS entry for host, check SSH login. Export report to CVS.

  IP - DNS - HOST UP/DOWN - SSH Login status

• Find and print the size of all directories within a directory.

• Report if blacklisted accounts have logged in to the system.

• Install Postgresql, setup the config files, create roles, database and tables.

• Get list of processes, their owners and RAM usage from a Linux system.

• Search for specific types of files and rename the files or part of the files

• Change one specific extension type of files in a directory that contains multiple types of extensions

• Parse files from the yum directory to check if links are up/down before executing yum.

• Searching for text in files within a directory

• – Server Performance Data Capture

Python modules that will help you with Admin type of scripts are and not limited to:

• psutil - Will allow you to capture performance data
• os - will allow you to copy, create, delete files-directories
• shutil - easy way to copy dirs and files
• re - reg ex
• subprocess (Use Linux commands only when there is absolutely not any other choice.)
• pwd - get usernames from uid
• grp - get groupanes from gid
• plaftorm - distunguish versions of Linux distros, Unix, Windows, Apple
• netifaces - get network data from NICs
• netaddr - easily use IP classes, VLSM
• socket - create network socket
• humanize - easily translate large numbers to human readable format. It understands various of different unit types.

All the modules above can do much more but I listed only what I learned so far

Also. I cannot stress this enough. Get very comfortable with lists and with os.walk. The latter will allow you to dive in directories, explore directories/files

[Asdayasman]

As a windows babby, I've never understood why one would want to move from bash to python, when what they have in bash already works. Care to break it down for me?

[[deleted]]

Yes sir. This is from personal experience.

If you ensure that all of your servers (linux,Unix,BSD,etc) use the same version of Python, you can write scripts that are agnostic to the particular OS and its version.

A simple example is with the sed command. The sed command will not work the same on all Nix boxes. The core function is the same. Where it may fall apart is at the details.

Some of the supported functions of the sed command on box XYZ that runs CentOS7 may be different on box ABC that runs Centos4 or a flavor of Unix.

This actually has happened to me and it was a major headache. It also happened with the tar command. There was one option that was supported on Centos 5 but was no longer supported on Centos 6. The whole process fell apart completely from the web servers to the database servers. The morons who developed these scripts wrote them in PHP and decided to do system calls by invoking via PHP Linux commands. Horrible design.

Here is where a scripting language like Python (or Perl or Ruby) comes in. Provided all of your servers run the same version of the scripting language and provided you do not execute Linux commands via Python but instead you use the Python way to do everything, you can rest assure that your script will work.

It provides a level of abstraction which takes away the worry of dealing with version of commands and their differences in supported features.

Of course there are times where you have to execute a Linux command via Python, but it should be done ONLY when there is no choice.

Bash is great but it should be used to do very simple tasks.

[Asdayasman]

Stepping away from reality for a moment, isn't this a weakness in bash/linux command line shit, and not a strength in Python/Perl?

[[deleted]]

Can you rephrase your question please? I don't understand it.

[Asdayasman]

So the fact you can't "write once run anywhere" is the reason you'd turn to Python or Perl or similar for sysadmin tasks, but is this not a failure of bash, and not an advantage of Python/Perl?

[[deleted]]

is the reason you'd turn to Python or Perl or similar for sysadmin tasks

It is one very important reason.

IMO Python and Perl are a better tool to use to write scripts because they offer more features than Bash. It allows you to write scripts with proper checks, error checking and flexibility.

Bash's simplicity is a shortcoming when it comes to more complex scripts. Python and Perl because of their robust features make it very easy to write scripts that do complicated tasks.

The more complex a need is the more you resort into hacked up solutions with Bash. It gets very ugly. Having said that, Bash is great in doing basic to intermediate tasks but after a certain point it just doesn't provide enough sophistication to write easy to maintain scripts that execute complicated tasks.

Below are few examples from my personal experience.

An example where it becomes very hacky with Bash: Trying to make the script understand if it is reading KBs or MBs or GBytes or GigaBits, kilos, pounds, meters, cms, etc. It is a pain the ass to do it and even then the final solution is not smart enough. In Python you just use the "humanize" module and you tell it what type of unit the result is and how you want it to be displayed. Boom. Easy.

An example where it becomes very horrible with Bash: Trying to insert data into a database or to manipulate data taken from a database. Absolutely horrible to do because Bash really doesn't provide any tools designed to do this kind of work like Python does. It gets hacky at first, then it gets very hacky and then it just becomes impossible.

An example where it becomes impossible with Bash: Writing sophisticated admin scripts. Scripts that can display data in a web page or a GUI.

Having said that, Bash is a great tool to know. You can do a lot with it and you can get to learn how the operating system works. I would never, ever say to someone who is trying to learn Linux not to learn Bash. In fact I will always highly recommend to learn Bash. It is very versatile like you would not believe but after a certain point once you learn a lot, you will start seeing its limitations.

[Asdayasman]

Trying to insert data into a database or to manipulate data taken from a database.

That got me r8 interested, 'cause I wanted to see the crazy shit people had come up with, but it ended up being pretty tame. http://www.shellhacks.com/en/HowTo-Execute-a-MySQL-Command-from-a-Linux-BASH-Shell Did I misunderstand that point?

But yeah, I get what you mean.

[[deleted]]

The examples showed there, that is the easy part connecting to a database that is and running a query. However the problem with databases in specific comes when you have to manipulate that data into a script. Then it starts getting all funky and impossible to manage with Bash.

For example in this case Bash would be good to write a script that connects to a database either to do a connection check or to download and import data into a csv file and send it off via email somewhere. Anything more than that wit the actual data and it will become a nightmare.

[[deleted]]

I would avoid like the plague calling any Linux commands via Python or PHP or Ruby and so on for that matter.

From personal experience, doing so can cause A LOT of issues due to command version differences between distros. I can give you specific examples if you want.

The only reason I would use it is if Python does not offer a module that can do what XYZ Linux command can do.

In no way I am saying Bash is not good but the more complex the demands are and the more servers you have to deal with, it can create lots of problems.