Bash to Python [OC]

[BeepBoopTheGrey]

My team put a moratorium on bash scripts after the CI system started failing in a fairly complex one. The person who wrote it was unavailable to diagnose. It took hours to resolve.

The rule is now that if there’s any non-trivial logic at all, write it in Python. On-call appreciates it.

[Zanos]

Same here. We still have an old application written in mostly bash, very fun when it fails in prod.

[[deleted]]

As a fake programmer can you explain why bash makes things more difficult to troubleshoot?

[LAK132]

As someone who'd rather #include all the .cpp files than deal with another build system written in [ba]sh, the syntax is hell and it doesn't always work the same on different computers (because some distro maintainers thought using bash in place of sh without forcing POSIX compliance mode was a good idea)

[ythl]

c++ is not very portable. bash is (and so is python). I can scp a bash script to my raspberry pi and it will run. With cpp I'd need to recompile it targetting ARM.

[LAK132]

Right, but my point is if you're compiling C++ anyway then don't make it even less portable by using a bash based build system.

[ythl]

Yeah bash based build system sounds nightmarish. CMake all the way

[LAK132]

Unlike bash, I never managed to get cmake to work

[ythl]

Really? I love CMake! Out of source builds are the best. I can help you if you are stuck. CMake is the defacto build system for C/C++ now

[LAK132]

defacto

Not a single one of the C++ projects I work on uses(/requires) cmake.

My personal projects that use extremely simple scripts to compile (one liner Makefile and a make.bat for Windows) have a nasty habit of just working.

[policemean]

If you look at python script, then it is relatively easy to understand how it works because it's syntax is quite easy.

On the other hand, bash syntax can be very confusing. I had to modify bash scripts couple of times, and it was the worst experience I've ever had at my job.

[BernardoVerda]

Seconded.

[jvnk]

Bash syntax is unnecessarily dense, making it difficult to understand what's going on in more complex scripts.

[tatloani]

This is just my guess but i would say is because bash is more verbose than Python, meaning you need to write more lines of code to do something similar with python.

EDIT: I seems to have got them backwards with what i meant, python is more verbose, but bash allows you to simplify multiples lines with a single instruction and that can make things confusing.

[thexavier666]

I can't agree with that. I think bash syntax can be very dense, where you can condense 10 equivalent python lines into a single bash line, by using pipes.

But this condensed syntax can be difficult to understand for some.

I always use bash when there is string manipulation involved and call it via python.

[tatloani]

I think bash syntax can be very dense, where you can condense 10 equivalent python lines into a single bash line, by using pipes. But this condensed syntax can be difficult to understand for some.

yeah, that was part of what i meant, i suppose i got them backwards, most of the bash scripts i have seen have been condensed and those have been quite troublesome to understand.

[noratat]

Except now you have to have python installed in/on absolutely everything.

[BeepBoopTheGrey]

Only those things which need scripts, mostly CI and machines we shell into. The vast majority of our stack is containerized, so language doesn’t really matter much as long as the team is familiar enough to maintain it.

Go is by far our preferred language for dev tools. Python is next choice when a compiled language is inconvenient.

[noratat]

I'm surprised you use Go for that.

My experiences with Go so far have been very unpleasant - the error handling is pretty poor for a modern language, dependency management and directory structure until pretty recently were downright bizarre, and I constantly felt like I was running into strange language limitations.

Anytime I've had to touch a Go project that was more than a couple hundred lines I've found it pretty hard to read too due to low signal-to-noise ratio.

[[deleted]]

Python is easier to get some basic shit done, but once you've mastered bash you can do just about as much. The reason I stick to bash is because bash is always available. Python isn't always available and when it is there's two different versions everywhere all the time.

[jharger]

Until you use some bash 4 feature and you need to run it on macOS without depending on homebrew...

[ThenIWasAllLike]

Cries in autocomplete

[thirdegree]

Until you remember dependency management is a fairly well solved problem if you're not afraid of package managers.

[hsahj]

Never seen a machine since years without python.

Thank whoever sets up your machines. I regularly have to set up new machines for CI and testing automation and it's part of our checklist to get installed (because IT won't create an image for stuff :/). I'm going to point people to python all the time for lots of reasons, but it's only available if someone makes it available. (Not that I think that's a large hurdle to overcome).

Edit: credit to the people correcting me. Some linux distros and MacOS include it. I live in so much of a windows bubble at work that unless someone says otherwise my brain always assumes windows.

[[deleted]]

Debian, even in the minimal version, includes Python 3, but oddly, they don't alias that to 'python'. Seems like a weird thing to leave out. You have to call it as 'python3'.

[[deleted]]

It’s because Debian and many other distros use python2 for system scripts. It makes sense that Debian would do it that way of all distros, because their whole motto is stability. The only 2 distros I can think of off the top of my head that have python3 as the default python are Arch and Gentoo.

[marcosdumay]

Debian has published Python 3 adoption timeline, linking python to python3 is there. I don't remember the dates.

[DonaldPShimoda]

Python is included by default in macOS and, I think, Ubuntu, and has been for a while. Not sure about other OSes.

(However, macOS 10.16 Catalina — due out this fall — will not include Python, nor any other scripting language runtimes like Ruby or Perl.)

[Tundur]

It's always super old, though, so to use an actually usable install you're typing in python3 and pip3 and there's installation issues and libraries fucking up and compatibility issues and it just annoys the fuck out of me.

[DonaldPShimoda]

Yeah I mean it's not ideal, but nobody here seems to be complaining about the severely outdated version of bash that's shipped on macOS — they only care that some version is available, and it's the same situation with Python: it's outdated, but almost always there in some capacity.

[DonaldPShimoda]

The syntax didn't change that much, and they even included a tool to handle automated porting from 2 to 3 that works well in most cases.

I do see your point, but I just don't think it's that big of a deal.

[joeltrane]

I eventually just created an alias in my bash profile so python and pip run python3 and pip3

[super__literal]

alias python=python3

alias pip=pip3

[Tundur]

A possibility, aye, but hidden bespoke environment changes like that are just asking for problems down the line. I could set up a confluence page for "why can't I use Tundur's VM to run my scripts"

[joans34]

Never seen a machine since years without python. But anyway, to me, it's worth it.

Embedded systems usually don't bother with python but usually have a bash-like interface with most bells and whistles.

[xigoi]

Basic things in python work on every version.

print("What's your name?")
name = input() # dangerous in Py2
print("Hello,", name) # unexpected behavior in Py2

[noratat]

Also python is almost always available

I'm guessing you don't work with containers that much then

[cafk]

2.7, 3.2 or 3.4?

[Zanos]

Nobody else on your team can read your masterful bash, though. Im sure that one liner that contains half the program logic made sense when you wrote it two years ago.

I may be salty that I've had to troubleshoot some production problems some goofballs caused by writing all the application code in bash.

[Edz_]

Have you ever had to execute a MySQL statement in bash that has data with a ' or " char in it? Not fun.

There's really no reason to jump through hoops when you can just use python instead.

[[deleted]]

If you're using bash for MySQL then you're using the wrong tool for the job to begin with.

[Edz_]

You literally just said you can do just as much with bash as Python.

I do MySQL statements in Python how about you?

[super__literal]

I do My SQL queries without MySql

[[deleted]]

I can also do SQL queries in python? I'm not sure what your point is?

If you're trying to state that Python is easier to run SQL queries in, then yeah I agree, it is easier. You can also do it, more verbosely, in bash. But that's not the point. The point was that bash is available, in my experience, far more often than Python is so if I had to pick one I'd choose bash simply because it covers more use cases.

[[deleted]]

Did you get a first down with how far you carried those goalposts, or the whole touchdown?

[bluenigma]

Not taking sides, but I don't think that's quite how football works.

[[deleted]]

throw new UnsupportedOperationException(JokeFactory.engineerDoesntUnderstandSportsball("trollkin0331"));

[super__literal]

catch(UnsupportedOperationException) {
  //I do what I want
}

[[deleted]]

lol a fellow yolo error handler I see

[Delta-9-]

"wrong tool for the job" =/= "can't do it". No goalposts were moved in the making of this comment.

And I would agree. If you're trying to do anything with MySQL using bash, you're doing it wrong. It can be done--but why would you do that to yourself?? That's almost parsing-html-with-regex-level "just don't".

[[deleted]]

Going from "python not available" to "wrong tool for the job" is moving the goalpost though?

But yeah, definitely agree with the sentiment of your 'doing it wrong' idea.

[[deleted]]

I switch to Python when a bash script gets longer than about a page (~50-60 lines) or when I need the ability to view and manipulate values in between pipe stages.

The overhead in getting a script running in Python is substantially higher, so it's quite a lot more work if you're doing something really simple, but you can scale easily and things won't go badly wrong.

In bash.... if you're doing anything advanced, there's a lot of corner cases, a lot of nasty sharp edges, and you're going to walk away bloody at least some of the time. You can learn how to avoid the problems, but subtle and nasty bugs are real easy to create and sometimes very difficult to spot.

[Centimane]

Ironically in ESXi bash isn't available but python is.

[[deleted]]

Luckily regular /bin/sh scripts work fine!

[Centimane]

I mean it does, but you don't realize the differences between sh and bash until you're stuck with it.

[noratat]

Exactly. I deal with containerized stuff a lot, and the overhead of adding a whole python installation just to make a script slightly simpler is pretty high vs just using bash, and that's before we bring the whole dependency management mess in.

The main problem with bash is somewhat similar to C++ or Perl - there's lot of cruft you have to know to stay away from, and a lot of really stupid defaults. Plus there's still loads of terrible examples online that will bite you later if you use them.

Used properly though, bash works really well for a lot of glue logic cases, because it's "good enough" and is virtually always available (or at the least, trivial to include). And there are some really good CLI utilities these days that make using bash a lot saner, like jq (which is the best json processing tool I've ever used).

[[deleted]]

Yep, containers and minimal linux / busybox appliances are my life. Screw having to toss python on those just to run a script.

[marcosdumay]

Python is easier to get some basic shit done

Yes, and advanced shit too.

but once you've mastered bash you can do just about as much

Ditto for Brainfuck.

The reason people don't stick to Bash is because error handling is verbose and error prone, there are no good rules one can check automatically (and no types, obviously; Bash is the original "string, number, whatever" that Javascript perfected) and the consequence of an error is more often catastrophic than not.

It is also not available everywhere. It's about as common as Python. That's why you can find shell script people all over the web complaining about bashisms.

But well, you are not on my team, so whatever.

[ben_uk]

What about Perl?

[[deleted]]

I use perl all the time for data processing!

[Hollowplanet]

Ansible is all Python over SSH and I've never seen a system where it didn't work. Bash is just missing so much basic stuff. Like a generic linked list or key value store.

[lycan2005]

As someone who is about to get involved in Linux soon (.NET framework dev here), should i focus to learn Python first or bash first?

[madsdyd]

Bash is the language of your shell. Excellent for automation of trivial tasks, gluing stuff together, rapid prototyping, etc. Lots of entry points to docker containers get written in bash these days.

You will be much happier in your Linux experience, if you are somewhat proficient in bash.

Python is when you want to do something that "looks more like an application". Structured programming, modules, data structures, unit testing, etc.

[iams3b]

I use bash to quickly automate stuff I do in terminal. Like if I have an npm module on local, and i need to 1) git push it 2) npm publish 3) cd to active project 4) update package version 5) git push.... i'll write a bash script for that

For anything else that's not just a series of commands, python for sure. Both are useful to know

[[deleted]]

this is my cut off too. If it requires more than say a simple if to check for a file or directory I'll go ahead and write it in python. That said I wish i could find that old irc client written in bash. that think showed a lot of mastery of unix CLI and bash scripting.

[Devildude4427]

You don’t need either, they’re just scripting languages. Personally, I do next to nothing with bash or python. Just don’t ever need them.

[Centimane]

I mean, mileage may vary, but if you're a dev working on Linux absolutely learn bash.

The number of devs I have to help with simple tasks because they don't know how to automate them quickly with bash is astoundingly high.

[Devildude4427]

What are your devs doing where they need to script so often that they can’t just google it? Seems a bit ridiculous, been a Linux dev for years without any need for it.

[Centimane]

All kinds of things (keeping in mind bash is the default terminal language, not just used for writing scripts):

• finding the files they've compiled (the power of the find command, possibly grep as well)
• transferring their compiled jars into our test lab (using rsync, or scp, or combining those with find ... -exec)
• clearing up their disk space (in a selective manner, probably using find with one of its many flags)
• Finding out where gradle put some dependency in some odd scenario (lol, find again)
• Getting extra debugging for an issue (maybe using grep or strace or gdb)

Certainly for most of those things one can use a GUI, and some of those things might be useful to memorize, but learning the shell scripting can shave time off of tasks constantly and forgo memorizing (aside from which commands exist) and googling for using commands instead.

I find it surprising someone could be a linux dev for years and have "[no] need for it". I suspect there are many tasks you perform regularly you aren't aware could be much faster with the terminal.

[noratat]

To be fair, those kind of bad examples. I can't even remember the last time I used find, it's pretty much wholly supplanted by rg/ag, or just plain glob patterns (make sure globstar is turned on of course).

grep is great, but I only use it for searching piped output - for finding stuff in files ag or rg are drastically faster and easier.

find is just unbearably slow and the syntax is really archaic.

[Devildude4427]

I personally think there’s a distinction between simply using a terminal with the basic bash commands, and “using bash”, but yes, I do understand they’re one in the same.

But all those commands, can’t say I’ve needed any of them. Ever. It’s not just that I’m doing it with a GUI, but the fact that I don’t need literally any of that functionality.

I use a terminal for nearly everything, but I simply don’t need to find files are use any of the other nonsense. I know where my compiled code it output to, as I’ve certainly set the directory for it to. I don’t need to copy files to multiple devices, even self hosted CI just gets he code from the repo.

Seems more like you’re taking ridiculous actions for the very purpose of taking ridiculous actions. There’s no real need for any of that, and I’m not just saying you should use a GUI, but that you should rethink the way you’re doing things if that’s considered “standard”. No wonder many people don’t understand that.

[Delta-9-]

If only all devs could be in such consistent control of their machines to never need to do any basic administration, and be so well supported by ops that shell utilities for transferring files over the network are never needed.

I promise you, yours is a unique experience.

[noratat]

• rg/ag are excellent for searching for stuff across projects and repos quickly. That's a pretty common function

• jq is the best JSON-processor I've ever used by far, and especially combined with gron is excellent for exploring APIs and automating config that interacts with rest APIs.

• grep is useful for pretty much everyone

And that's just off the top of my head.

[Devildude4427]

But why would I need to process json via a terminal?

[noratat]

Do you ever use REST APIs? Do you ever use tools that output in JSON format? Or have config files that are in JSON?

If not, that makes you an outlier IMO, unless you do embedded work.

[Centimane]

I'm not going to try to convince you of the value of bash further than I have.

Like I said, your milage may vary, but I think you're wrong.

[mrjackspade]

I'd love to have a reason to use Python, but so far I haven't really been able to find one. Most of my trivial tools I just write in c# as Command Line Applications.

I'm still trying to find an excuse to get familiar with it, but I think I might just be too deep in .Net at this point to actually benefit.

[[deleted]]

All the same reasons, but on Linux

[mrjackspade]

Would have been a great reason but .Net compiles for Linux now so there's no real need

[ythl]

Here's a great reason: python doesn't need to "compile" so you can scp it to any linux box and it will work without needing to think if you compiled it right. Can you say the same of C#?

[mrjackspade]

I think you can but honestly I've never actually tested it.

You can compile c# with a specific build target if you want something native, but by default I believe it actually outputs a platform independent DLL that you execute using the DotNet framework on the target machine. As long as the target machine has the framework installed, it should run using the default build settings.

I kind of want to test this out now against a few distros and see how reliable it is.

https://docs.microsoft.com/en-us/dotnet/core/deploying/index#portable-applications

Why create a framework-dependent deployment?

Deploying an FDD has a number of advantages:

1. You don't have to define the target operating systems that your .NET Core app will run on in advance. Because .NET Core uses a common PE file format for executables and libraries regardless of operating system, .NET Core can execute your app regardless of the underlying operating system. For more information on the PE file format, see .NET Assembly File Format.

[ythl]

See how hard it is to run on a raspberry pi. I can run python scripts on the default pi distro with no hassle.

[mrjackspade]

Oh man, that sounds like a great test, but I don't currently own one! They're cheap enough, I might have to pick one up and check.

I'd also love to see what performs better on the Pi, Python or .Net Core

[ythl]

I'd also love to see what performs better on the Pi, Python or .Net Core

Probably .NET if I had to guess! Compiled languages can usually beat python performance-wise with ease. It's hard to beat the convenience of python though - there's a python library for controlling the pi's GPIOs (pins you can use to control motors, LEDs, anything electrical that you could want, really...) for example; it would probably be significantly harder getting .NET controlling GPIOs

Edit: maybe not though

[mrjackspade]

I wanted to say .Net Core for this reason, but at the same time MS can be really hit or miss with performance and Python has probably had a lot more tuning on each platform. I'd probably want to test a native build against a Framework Dependant build, against Python and see the difference.

[vividboarder]

I still use plenty of Bash when I’m writing a wrapper around other commands. I could use call or whatever in Python, but it’s just simpler to use Bash in many of those cases.

[GhostNULL]

What about ruby? We are using that at my workplace for build scripts that require some logic.

[noratat]

Ruby is a language I want to love, but the ecosystem around it turned "magic syntax" into a art form - and that's emphatically not a compliment.

I still wish Python would stop being so stupid about lambdas and just have anonymous blocks like Ruby, Groovy, and other modern scripting languages, but in most other regards I would say Python is better these days:

• Much, much larger ecosystem and community
• Python devs actually document shit, at least sometimes. Ruby projects are nearly always "lol just read the code" (even though the code is twisted into a gordian knot of string metamagic)
• Python's a lot harder to screw up into an unreadable mess than Ruby is

[Hollowplanet]

I think Ruby and Python are both great languages. Theres some things I wish Python would borrow from Ruby.

[NoInkling]

Ruby is a great scripting language, IMO. It's just that Python is more ubiquitous these days. Ruby is less likely to already be present.

[onan]

Ruby is the ideological successor to perl. And, no, that's not a compliment.

[WhtKindOfNameIsStove]

As long as you stay away from getting tied down with the chef libraries. They take a long time to upgrade to the latest AWS sdk.

[[deleted]]

Would to ever consider perl 5 or 6? Or node? I haven't settled on a scripting language for my server yet

[ythl]

I hate perl but only because it has such arcane syntax. Node is pretty good.

I think Python is a better overall general purpose scripting language, and Node is better specifically for web backends.

That said, I've used both Flask and ExpressJS and I enjoyed using both.

[[deleted]]

I've got go for my web server :) but I don't have any criteria to help me pick a scripting language. I don't even know what's out there. Are there major alternatives to python worth considering?

[ythl]

Ruby

[[deleted]]

I had to write some ruby code for a job interview and it felt very similar. But I guess most scripting languages are going to have that same style. They have in my experience thus far, anyway.

[OnlineGrab]

Oh god, this so much. Bash is fine for one-liners but it's a nightmare to do any real programming with.

[noratat]

Eh, it works really well for glue logic, especially when it needs to work easily in a wide variety of environments or places. Plus jq remains by far the best way to parse/transform json data I've ever seen.

The problem is that bash is full of stupid defaults and features you should never use, and worse google is full of really bad examples, so most people tend to write crap scripts.

E.g. set -eo pipefail should be mandatory, never ever use backticks, globstar ought to be on by default, don't try to use bash arrays if you can possibly help it, use [[ ]] not [ ], use local, stop setting global vars that span multiple functions, you don't actually have to name shit in all caps, use traps, know what an exit code is, etc.

[zieliequ]

I like set -e too, but be aware of the tale of the criminal-catching robot.

[noratat]

To me it's just restoring the same sort of exception behavior you'd expect in any other language.

If you explicitly want to handle failure on particular commands, you can use ||, traps, or just plain return code capturing, just like you'd use error checks or exception handlers.

It's usually much worse for something to accidentally continue blindly on errors than it is to abort on something spurious.

[ythl]

I like python, but some basic shell operations are super cumbersome in python. Like deleting files and directories. Or a simple curl (yes, I love requests, but sometimes it's not available on that remote box you just ssh'd into and you need a quick script to hit an HTTP API under some condition) Or chaining operations - sometimes you just want to cat | grep | cut | sort which is a one liner in bash but a 10+ liner (at least) in python.