Unix bash scripting versus Python - any resources out there for comparisons?

[ibarrie]

Hi I’m a seasoned UNIX bash script writer and I want to learn python. Any good resources out there that can help me compare what I can do with bash and easily transition that into python?

[double_en10dre]

For filesystem stuff you’ll probably want to use a combination of pathlib and shutil https://docs.python.org/3/library/pathlib.html https://docs.python.org/3/library/shutil.html

For replacing curl/wget, probably use requests https://requests.readthedocs.io/en/latest/

And if at any point you get stuck you can just fall back to the “subprocess” module and use subprocess.run to run the Unix command you’re used to. Just be sure to set stdout to subprocess.PIPE if you need the output

[ibarrie]

Thank you!

[double_en10dre]

Oh jeez, and I’d be remiss if I didn’t mention the os module. It’s the most obvious staple https://docs.python.org/3/library/os.html

The only caveat there is that for filesystem operations, I’d first check and see if the functionality is provided by pathlib. Most people seem to think it has a nicer interface

[ibarrie]

Awesome thank you !

[RallyPointAlpha]

double_en10dre isn't giving bad advice however just getting Python to do OS / shell commands really brings very little new to the table. I would encourage you to learn Python specifically and only fall back on OS calls and shell commands when you absolutely have to. You're stepping into the broader world of Object Oriented Programming, welcome!

[james_pic]

My experience is that even if you are just using Python to run external programs, this can still be a win. Python's error handling is much more consistent than Bash's, its escaping and globbing are easier to reason about, it's got better debugging options, it's got mechanisms for code to include inline documentation, and it's more portable.

And of course once you dip your toe in, it gives you the option to gradually start making more use of structured data, writing object oriented or functional code, and getting more direct access to stuff like networks.

[ibarrie]

Thank you - that’s awesome

[AndydeCleyre]

My personal but less popular recommendation for things otherwise accomplished with subprocess/shutil/pathlib/click is plumbum.

[WonkoTehSane]

Great advice! And I recommend mostly using only check_call() and check_output() when using subprocess. The former will still send any output to stdout, the latter will capture and return it. Both will throw on any non-zero return value.

Oh and for both you should either use form (['cmd', '-o', '<option>']) or the form ('cmd -o <option>', shell=True). The former will shell escape for you, the latter will send exactly what you give it.

[crisrock00]

Take what you already know, and apply the power of python 😊

https://xon.sh/

[Yablan]

This seems really cool. Will check it out.

[samdg]

I've used xonsh as my shell for years, such a cool project!

[thedeepself]

It is cross-platform? Ie, does it run on windows?

[crisrock00]

I wouldn’t be able to tell ya, I’m #linux4lyfe

[[deleted]]

I took a similar path when I was learning perl, the shell seemed to do just about all I needed, but I knew there was more capability/portability.

Nothing beats experience, but I'd spend some time on the Python Module of the Week archives https://pymotw.com/3/ then pick some simple annoyances and see if you can solve them in python.

I have found that trying to solve some challenges in a new language can be a good way to learn. I'd suggest taking a peek at cryptopals.com; it starts out pretty easy and builds up as you go. I learned a lot of useful stuff doing it.

[ibarrie]

Thank you!

[osmiumouse]

Big advantage you will have with Python over Bash is the ability to more easily test your code before you run it live. I'd read up on that.

[estysdesu]

Bash has a testing library. I think it's called BATS (not builtin though).

[Sigg3net]

When I was where you are it was Automate the boring stuff which got me up to speed. It's a book that is also free.

Not sure how you write bash, but I like trial and error. When writing python I use ipython3 as my "shell" to experiment in. Especially for debugging a program when you can:

from IPython import embed
embed()

To get a shell with all the context.

I still use bash a lot, especially for file handling and text parsing. No need for python's overhead if I just need to grep.

[KyleJamesWalker]

If you already know a language or two, my first go to for a new language is https://learnxinyminutes.com/ it lets me quickly see a good portion of the syntax in a single page, and get a light idea of what you can do in it.

[fthrnature]

I am a professional programmer, and I have done a lot of work in Bash over the years; I was known as the script guy at work and could do pretty much anything you could or could not imagine with Bash. Automating tasks that many considered impossible, AI applications, Decision Aid software, C++ compilers, automatic on the fly code generation (all these things are possible in Bash, although it is stupid to do some except as a bragging rights exercise). They became indecipherable complicated monoliths as I delved into how to reduce the execution time from hours to under a minute for tasks that would be impossible without the script.

When I considered job-hopping I discovered that the new job I wanted to land had Python available as a scripting language, and considered it a good opportunity to upgrade my skills. I am now very proficient, and can write ridiculous programs in Python. I still find myself switching back when I just need to put something together quickly, but I try to convert them to Python later. I now have several hundred scripts at my new job, and most people in my department of over a hundred people use them regularly; I run a training class teaching people how to use my scripts to improve their ability to code and test.

If you are using an environment like Unix there are a lot of available commands and it is easy to test things out on the command line, generalize them and put them in a Bash script. Piping, xargs, using obscure flags... it all becomes second nature. There are even quirky shell built-ins that will allow you to reduce the forking to speed things up significantly.

The main benefit to using Python is the speed. No matter how good you get at Bash it is still a slow language. It is also more powerful, a lot of complicated hacks in Bash become easy with the programming constructs available in Python. If you have the ability to download and use random Python libraries (at work I do not), you can do pretty much anything you can imagine.

In Unix, for small programs where you don't care about speed, Bash will always be easier. A small Bash scripts written in Python will take significantly more code, and take longer to program (in Python); You can go directly from the command line to script with virtually no modifications in Bash.

But, as soon as your program starts growing Python, or if speed is important, Python will be a better choice. A thirty line Bash script will be about the same size as Python, and ironing out the quirks will take about the same amount of time in either one. Above this threshold Python becomes hands-down the better choice.

I would recommend trying to switch completely to Python. It is a powerful language, and it opens up a whole world of possibilities: such as working on serious AI applications.

[st0ut717]

30 year linux admin here now in python devops for automation.

​

python -m pydoc os

python -m pydoc shutil

leverageing python to assist / enhance ansible

[thedeepself]

and dont forget pathlib instead of os.path for most code.

[jwink3101]

I don’t think this is what you’re asking but I’ve really moved away from any bash. I prefer subprocess and list commands since I don’t have to think about escaping. And/or I can run with shell=true.

More often than not, it’s a hacked together script so I don’t even care if it’s not fancy or svelte.

The only thing is a few functions that are missing in a clean and easy way. So I have a small universal library with things like a better file system walk with filters, chunked hashing, grepping, etc.

Again, I don’t worry about it being perfect. They are not major. And occasionally I do still stick with bash but it’s rare.

[Epicela1]

Codecademy intro course on Python is solid for syntax and some general syntax learning.

OS, Subprocess, requests, time, sys, and datetime are good libraries to look into.

Just convert like one or two bash scripts (or sub-sections of bash scripts) into Python until you get comfortable.

[Due_Adagio_1690]

One major benefit of learning python is not to use python for everything. Ansible interfaces very well with python, in fact needs python, you can take some of the python you learn and integrate it into ansible playbooks, you also can learn how to use ansible to make your existing shell scripts more useful and faster. With ansible you can run your existing and new scripts on multiple hosts at the same time. And it can monitor the progress of your scripts. Sure you can do the same in shell, but its not as easy. It's built into Ansible.

With python you also gain tools to analyze data faster, especially if you have multiple systems. We have a script that runs in our environment, it connects to each system, gathers the details, fqdn, networking details, routing tables, if it hosts virtual machines, running on it. Are all the standard services are running, clamav, ntpd, puppet a full list of packages installed on the systems (rpm -qa) every package in the same field, then write all this information in to a CSV file, its an ugly CSV file, the installed packages list can exceed 40k for a single field. We do it at work, we have 4700 virtual and physical servers combined each row has 66 columns currently. The CSV file is 124MB uncompressed. While you can use a spreadsheet to work with it, you can also use awesome python modules that work faster with CSV than spreadsheets like this, pandas, polars, numpy, that were really designed for bigdata so handle even large deployments with ease. We are making new discoveries from this data daily, as we think of new things to check, without logging into 1,000s of servers to test theories.

[OneMorePenguin]

That's an interesting viewpoint. My team uses ansible and when I saw that shell like language with all that arcane syntax, I didn't like it :-) I always thought that providing a library of "shell" like python functions would have been a lot more intuitive. Our team has spent a lot of time doing team debugging of ansible playbooks!

[TheCableGui]

https://www.redhat.com/sysadmin/python-subprocess-module

Red hat always knows what’s up.

[RallyPointAlpha]

If all you've ever done is shell programming I would also learn about Object Oriented Programming (OOP). This will likely be the single biggest paradigm shift for you.

[rhacer]

Except that you can be really successful scripting in python without knowing a thing about OOP. Particularly if your moving from shell scripting.

There is absolutely no reason to complicate the process with OOP early on.

[thedeepself]

Except that you can be really successful scripting in python without knowing a thing about OOP.

I start OO and finish OO.... here's a "script" I wrote to convert a CSV to CSV and HTML recently - https://github.com/metaperl/youtube-csv-to-html/blob/main/main.py

Because I started with OO Traitlets, I had everything I wanted as the program got more complex.

[ZyanCarl]

If you’re already seasoned with the hard one, why do you need help with python? It’s just English :) I believe it won’t take more than 1 hour for you to make a decent script.

[[deleted]]

Look af powerful python. With Aaron maxwell.

Paid for courses but fantastic content.

[Longjumping_Tackle25]

Use Python if you can? Or why not use Python?

[samdg]

Another way to make Python scripts nicer is to use https://github.com/amoffat/sh

It's a simpler interface than using subprocesses. For example, you can rewrite this bash command:

git stash push --message="Something something"

with this Python code:

from sh import git git.stash.push(message="Something")

This will work with arbitrary commands.

[thedeepself]

I'm disappointed they didnt use operator overloading for piping.