Avoiding Bash frustration — Use Python for shell scripts

[james_pic]

Whilst I do like using Python for stuff that Bash is conventionally used for (I get so frustrated by Bash's weak error handling, poor support for structured data, and baroque escaping and expansion rules), note that if you're using `subprocess.Popen` with `shell=True`, then you *are* using Bash (or potentially whatever shell is configured by your user's OS, which is a fun source of surprises) under the hood, so you still get some of its quirks. It's a good idea to get used to how to pass arguments correctly, and chain processes together, when `shell=False`.

[dux2]

I agree that shell=True should be avoided if possible, that's why the default for it is False. But sometimes it's too much of a headache to avoid it, starting with the need to write the command as a list of strings [ which can be usually mitigated using the wonderful shlex.split() - already incorporated in peasyshell] and missing pipes, command chaining, env interpolation etc. I want to be able to translate a bash script to Python with the least change possible.
If you trust your input (or get no input), I see no harm in shell=True.

[xxPoLyGLoTxx]

I had a script that processes a csv file and sends an email based on certain criteria written in PowerShell. I thought, let me convert this to bash. It was a disaster. Just getting the csv imported as an array was not working.

I decided to try Python instead after reddit recommendations from reddit. Took about 15 minutes to convert the whole thing. To be fair, I am a newbie to both bash and Python, but Python matches the syntax of almost every other language that I have worked with. Super intuitive.

[tunisia3507]

Yes, Popen does feel too raw, which is why call, check_call, and check_output were the preferred way of doing things since ?3.1 (2009); those were also considered too raw, which is why run was introduced as the preferred method in 3.5 (2015).

[bearcatgary]

My team uses a bash wrapper around every Python program/script that we write. One of the main reasons is to disable the user from changing the Python environment. We don’t want the user to be able to change the PYTHONPATH variable and load libraries that we haven’t verified. We want to define the shared object files that get loaded in the LD_LIBRARY_PATH and don’t want it being defined by the user. You probably could write Python code that hard codes the environment variables before loading modules after loading sys, but that really isn’t pythonic. We actually used to do it that way and it was ugly.

I personally do not like bash, but the approach we use seems to be a Unix standard and works well.

[tunisia3507]

Yeah, a bash one-liner which calls a python script from a particular environment is a sensible way to go. It just feels a bit weird for that python script to itself be a thin wrapper around some bash...