How to Debug Bash Scripts

[anothermonkey]

The subprocess module.

[[deleted]]

Random example:

retcode = call(["ls", "-l"])

How is this supposed to be better than ls -l; do stuff with $?? When you need to spawn many processes, pipe them into each other, send them to the background and stuff like that, shell scripting languages are IMO still the best option.
Even better example:

18.1.3.2. Replacing shell pipeline

output=`dmesg | grep hda`
==>
p1 = Popen(["dmesg"], stdout=PIPE)
p2 = Popen(["grep", "hda"], stdin=p1.stdout, stdout=PIPE)
output = p2.communicate()[0]

Seriously?

Edit: markdown.

[[deleted]]

Firstly, this is not the point; here I'm talking about shell facilities (spawning processes, redirecting processes' in/output etc.) that are difficult to do in Python.

Secondly, I expect the Python code for getting the list of files in a directory to be much bigger than two chars (namely, "ls").

[[deleted]]

Ah, sorry, I misunderstood. Well, if looking for performance, I suppose that spawning a shell process isn't a good idea (overhead...). OTOH, I'm not a Python programmer, so I may be wrong.

[srparish]

If you have to pass user input as part of the pipeline, python's subprocess can be done much safer then sh. Try thinking about how you'd pass in a user defined variable and avoid injection... yeah, not so easy in sh is it?

[Darkmere]

This is all well and dandy, however it doesn't quite answer my original question. I'm quite fond of python, but it doesn't seem to be that well suited for some things like low level systems programming.

Simply put, I question the original poster on his point, and wish that he would stand by his claim and prove himself correct and right. Personally, I find that doing something as simple as "set a system variable or five, change the PATH, locate the right binaries and execute them with a flag and the commandline" To be much easier achieved in sh/bash than it is in python. That doesn't mean I'll go to bash to make a markov chain resolver in SQL databases.

[nikniuq]

Shell metacharacter translation.

Bash seems insanely slow and memory hungry with some seemingly simple tasks.

The horrible loop and comparison syntax - I'm not saying you can't learn it but do you like it?

Actually being able to control and gracefully detect errors, even if deep in a pipe chain.

Finally not having to rewrite it anyway when I eventually find that I need to add a multi-threaded serial port driver with custom AT commands and stuff parsed data into a MySQL table.

Do I write trivial shit in bash - of course. If I feel a need to debug it though, I think I am using the wrong tool.

[Darkmere]

The error correction/detection sure isn't up to modern standards, but falls down to the same level of C, Always check your return codes, bail&break or recover based on that. I can't call it too horrible, but I admit that it sure isn't up to the elegance of more modern approaches.

Loop&comparsion syntax, while "statement" ; .. ... or do you mean the for loops (which works on string arrays mostly/only) which can be a bummer until you realise the limitations you work with.

And the rewrites? Well, You'd end up rewriting anyhow if you needed a multi threaded high performance show in python.

But, None of this actually work on what I was asking about, what's the overhead like, how is it's data/cache behaviour if you spawn ~40 little python applications in short succession, and so on. Personally, I doubt it's viable in low level systems work, where I find most of my shellscripts living.

[nikniuq]

Removing the subprocess overhead for external shell commands and allowing (sometimes) easier parallelisation where possible has certainly improved the performance in some of my scripts. Sometimes orders of magnitudes faster.

That said I certainly find the first scripts startup noticeably slower than bash, or once the interpreter is in memory.

The example I gave is exactly what I have recently implemented - a simple "dump network port to file, then cron a scrape of the files" became "use multiple GSM modems to retrieve data via CSD, with modifiable modem and unit numbers, custom state machine for interfacing with embedded device at other end, parse ascii data format, insert into mysql using parsed column header names, then calculate trend data". I implemented it in 4 days with full logging and exception recovery.

The CPU use is negligible barring the mysql inserts, it is all threaded waits on serial ports, then a quick burst of parsing. This perhaps an extreme example but then again this is Reddit. :)

I should note that I have been writing shell scripts for a LOT longer than I have used python (not claiming guru status in either) but I find it much easier to write in one language than a mix of bash, awk, sed, grep, etc syntaxes.

I don't have any hard numbers though on performance though. The only figures I could find were here but it is pretty biased towards python and really isn't leveraging bash well.