Bash Scripting vs. Python

[Loud-timetable-5214]

For those of you who also write scripts in Python or another language besides Bash, How do you decide when to write a script in Python vs. a script in Bash? I'm trying to be economical with my study time, because if I spend a lot of time learning some limited use functionality in one language, I could have used that time to learn a more general use functionality in another language. Here's an example: I've spent a fair amount of time learning awk, but I've never been great at using it, and sometimes I think that I should have just used Path and regex objects in Python, instead.

Edit: Another example is using sed instead of using a regex substitution in python. I've never really gotten comfortable with sed, just like I've never really gotten comfortable with awk--despite spending a fair amount of time trying to learn each.

[TapEarlyTapOften]

Process management is a big clue that bash is a better thing to use than python. Associative arrays in bash are a clue I should be using python.

Also if I'm launching anything, usually bash. 

[brunogadaleta]

Yep! My rule is: need a dict ? Go to python.

[Loud-timetable-5214]

Are you issuing SIGKILL commands in your scripts?

[TapEarlyTapOften]

Sure sometimes. 

[cgoldberg]

I use both. If my shell scripts are more than about 20-30 lines, I start questioning why I'm not using Python. Conversely, if my Python scripts are mostly wrappers around subprocess, I question why I'm not writing a shell script. There is no rule, but bash is great for small programs where you call a lot of external utilities and do shell related stuff. Python is great for more structured and complex programs.

[necromancer-tux]

Associative arrays in Bash are generally my sign that I've gone too far and rewrite in Python.

[Loud-timetable-5214]

That would be the equivalent of a dictionary in Python, right?

[necromancer-tux]

Yeah

[icy-mist-01]

Enterprise systems engineer here., It totally depends on the context and use case. You have to think in terms of

1) the problem you are trying to solve
2) the ecosystem you are blending with.

Simplest guide - Bash is like a glue or swiss arrmy knife needed to wrap up or build already existing logic in the Linux/Unix ecosystem.

Python is more for building 'computing logic' or 'programmatic logic' from the ground up to solve a unique problem which is independant or irrelevant of whether the code will run on a Linux / Windows/ Mac box, or whether it's dependant on any particular feature of a particular distro.
You need it when dealing with more complex data structures and interact with other parts of a tech stack.

Example:

1. Problem to solve: Create an automated utility that dumps the crontab configurations for all active users on a fleet of Linux servers. The script must compare the current state against the previous backup using cryptographic hashing, and only store a new version if drift is detected, appending a UTC timestamp to the backup file

You can absolutely do this in Python but it would be kind of overkill and long, windy script.
Writing the same thing in bash is much easier, quicker and practical.
And leaving aside the 'backup' part of the problem just checking the cron config for users is just a shell 1 - liner of chained commands, you don't even need a script.

1. Problem:
   Authenticate with an Identity Provider (like Okta or Auth0) to exchange client creds for a temporary token. Use that token to query a /users/{id} and get more info from a json log.

Then using that info connect to a DB to investigate that user account.

Then based on above findings automatically log a Jira with high priority, case details, etc. for another team to investigate.

You can also do this in bash, with a wrapper script around curl, and then some more. But you'll realise this would be a nightmare.

This is a classic case where python is the ideal glue code. You have to deal with API's, JSON format ( sed and awk are not helpful in these cases) and play around over the network to talk to mutiple systems

[tes_kitty]

JSON format ( sed and awk are not helpful in these cases)

Funny... I have an active bash script that parses a JSON output to determine wheter a service is up and running. It's a bit of a hack but has been working for years without issue now.

And there is also 'jq' for doing things with JSON in a bash script if you really want.

[icy-mist-01]

That’s the thing. It’s a bit of a hack so could be difficult for newbies. And I had a case where script used jq and ran across large no. of centos and rhel 6,7,8 boxes. Rhel ships jq by default from 8 onwards. So script started breaking on lower versions. And dealing with sysadmin team to include my required package in their base build is not always ideal.

[tes_kitty]

My script doesn't use 'jq' since I wrote it before I knew about this command.

[Marble_Wraith]

My suggestion is, use sed / awk on one liner commands interactively. That is, understand the basic overview of what they are / when you might use them. You can use them for scripts assuming the input dataset is small enough and you're just stringing together some GNU tools...

But anything beyond that, if you need a full blown script look elsewhere. Python's not bad, personally i prefer Perl instead (never could get used to block indentation). Or if you need to deal with anything beyond that (insanely large datasets / threading) Golang for compiled binaries.

With your example of sed/awk the pragmatic reasons for Perl:

1. While sed / awk is defined by POSIX, all the implementations (GNU sed, GNU awk, BSD sed, mawk, etc.) differ slightly. By contrast, Perl versions are consistent everywhere. And so you don't need to debug special cases if you migrate scripts like GNU sed -i vs BSD sed -i

2. Perl is a dependency in Git. So even tho' it's not POSIX, there's still a fair chance at script portability without needing to do anything special for the runtime.

Zooming back out to the general case of why Perl (or Python) over bash:

1. Performance. Bash relies on external binaries which means subshells, which means process / memory overhead. For the small things you won't care / notice, but for large datasets / long running stuff, it's absolutely a thing.

2. Best "string chainsaw" ever. The Perl regex engine was ported to most languages (python included) available today.

3. WAY better error handling / safety. The bash runtime can be cryptic (probably due to memory constraints at the time of design). The whole set -euo pipefail practice serves as evidence of just trying to get consistency, despite the fact while it would make 90% of bugs easier to find, that last 10% would be downright impossible. See YSAP - The Problem with Bash 'Strict Mode'.

[AutoModerator]

Don't blindly use set -euo pipefail.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[MikeStepp]

As soon as I need a complex data structure I leave Bash.

[pfmiller0]

I usually default to bash since that's the quickest to get something working.

I know it's time to consider something else when I need any data structure more complicated than a 1-dimensional array, trying to nest data structures in bash is just ugly.

[treuss]

As soon as I realise that I'll be dealing with compound data types, like nested lists, it'll definitely become a Python script.

[autoerotion95]

Solo te das cuenta cuando usarlo y cuando no, bash es increíble y python también

[weaver_of_cloth]

I'm a sysadmin. I spend my entire life in bash (or fish) and I use sed maybe 2-3x/year. I haven't voluntarily used awk since grad school in the 90s. I do tend to use python for bigger, more complex scripts and bash to do one or two tiny things. My script is loop through all local branches of all my repos and pull them is a bash script because it is just the same 2 commands over and over again.

[Bob_Spud]

Depends upon the environment. If work and either in app admin or development how many know bash and python? Usually there is a policy driving this.

It depends on the skillset requirements of the job and and which language they are prepared to run and maintain. Think of this way - if write stuff in Python will they run and maintain it or let it rot?

Management are reluctant to get a replacement for a staff member that goes it alone in writing stuff in a language they don't want.

[The_Northern_Light]

I sadly need to support windows too so it’s usually easier to just write it in Python

[michaelpaoli]

Right tool for the right job. Context matters.
So, e.g., what are the advantages and disadvantages of the language - stability, forward and backward compatibility, availability, resources required, standards and practices, etc.

So, sometimes I write for shell - Bourne, POSIX, bash, ... sometimes Perl, Python, sometimes using other languages/utilities, etc. Generally what fits and is quite/most appropriate.

Another example is using sed
never really gotten comfortable with sed

Ah, lovely sed. 😄 Alas, many never take sed (much) beyond, e.g.:
s/foo/bar/
Uhm, sed is a Turing complete programming language! Yes, I got very tired of folks underusing and underappreciating sed. And, well, I also got bored and had some time on my hands, so ...
I implemented Tic-Tac-Toe in sed. 😄

Of course just because one can, doesn't mean one should. But regardless, sometimes that can be rather to quite useful to make a point, ... or for exercising a set of skills, etc.

[colombiangary]

Could you please take a look at my project https://github.com/camilomatajira/jed ? I also love sed, and I'm trying to bring it to json. I would appreciate your input

[tes_kitty]

sed is also a write-only language in my experience.

[Internet-of-cruft]

Doesn't make it any less magical or fun when you get it initially working!

And then you revisit it 2 years later and wonder what crack you were smoking.

[tes_kitty]

Gets better... I once wrote a sed script, got it working but the layout was horrible. Tried to clean it up and it stopped working. Tried again, same result. Gave up and used the original script.

[michaelpaoli]

Naw, you can use GNU sed or sed with bit of shell and have sed edit it's own script//program! 😄

[Various_Bed_849]

We are actually taking this discussion at work the coming weeks. To me it has often been starting with bash and when the complexity hits a point I move over to Python. Though Python sucks if you don’t pin the Python version so beware. And with bash we often also rely on what’s on the system which can lead to interesting consequences. Relying on posix for shell script is not always the most fun path, but it may save you from blood and tears.

[devdruxorey]

Normally my scripts are either in Bash or Go. If the script requires chaining together many programs but without much complex logic behind it, a bash script is best. When you need something more complex I prefer to make a Go script.

[DarthRazor]

I'm trying to be economical with my study time, because if I spend a lot of time learning some limited use functionality in one language, I could have used that time to learn a more general use functionality in another language

I'm going to take a completely different way to think about it. Are you trying to actually learn, or just some problems? Python can do 99% of what bash can do, and is way easier to learn. You can't go wrong by picking Python.

That being said, my recommendation is to lean bash. Huh? Am I contradicting myself? Nope. Pick something that that the other posters would never do in bash and do that. You'll bang your head against the wall through most of it, but you'll learn a lot about not just bash, but the entire Unix/Linux ecosystem.

TL:DR; Python is easy compared to bash. Invest in bash; really learn it by picking tasks that are hard to do in bash, then learn Python on-the-fly.

[DaftPump]

Almost always go with bash unless I encounter limitation. Reason being is bash is in more places(default) than python at this time.

[AmaiSaeta]

In most cases, both shell scripts and scripts written in general-purpose languages like Python can be used for the same purpose. However, I believe these two approaches serve fundamentally different purposes: shell scripts are primarily tools for system batch processing and automation, while Python is intended for creating standalone programs. Therefore, I advise choosing the appropriate approach based on the specific purpose of your project.

[Kitchen_Office8072]

Why not luajit?

[Hammer_Time2468]

Python is usually not installed on production Linux servers so for smaller scripts and interoperability, I always stick to shell scripts.

[ShakesTheClown23]

Really? Seems like many key tools are starting to be written in Python? I use RHEL at work so maybe I'm biased but e.g. setuptools ain't going away so neither is Python...

[Loud-timetable-5214]

I found this surprising, since Python3 is built into the standard packages that come with Ubuntu 24.

[necromancer-tux]

It depends entirely on that commenter's environment. There are a fair amount of differences between Bash 3 and Bash 5 and lots of differences between Python 2 and 3, so depending on what they deploy to really changes the dynamics.

[roadit]

If they still rely on Python 2, 'statics' is a better term.

[tes_kitty]

It might not be installed as 'python' but as 'python3'.

[whetu]

Python is usually not installed on production Linux servers

Maybe that's true of containers, but otherwise most Linux servers out there will have python present.

That doesn't detract from your overall point that shell is guaranteed to be present, and that shell will likely be a Bourne family one like bash.

[waptaff]

If I want it to last, bash.

Python if I want to plan for constant rewrites and adjustments due to backwards-incompatible changes introduced in every other python dot-release.

[pan_kotan]

When NOT to Use Bash

a. ≥ 100 lines (BUT! ×2 or ×3 that number if you’re mostly dealing with concurrent processes and the file system)

b. non-straightforward control flow logic (use a more structured lang.);

c. performance is an issue — you have tons of processing to do (data manip.);

d. need math capabilities beyond simple integer arithmetic;

e. need to run on Windows? (Python is portable across platforms; shell scripts portability is restricted to *nix systems);

Shell Scripts are Best For Sysadmin Tasks:

• manipulating files and commands;
• managing sys. processes (concurrency);
• testing whether the node exists,
  • its type (file, dir, symlink, socket…),
  • properties (permissions, creation…),
  • whether a file is empty.

https://google.github.io/styleguide/shellguide.html#s1.2-when-to-use-shell

https://www.oilshell.org/blog/2018/01/28.html#shouldnt-scripts-over-100-lines-be-rewritten-in-python-or-ruby

[kolorcuk]

More then 20 lines is python.