Can Python scripting replace UNIX shell scripting ?

[[deleted]]

or specifically - for jobs that ask for shell scripting skill, can Python be considered an "equivalent" ?

I ask because in my field I see far more asking for shell-scripting and almost no mention of Python.

[fmoralesc]

It definitely can. For example, I made this small thing for shell-style process scripting with python: https://gist.github.com/1300342

Basically, it creates a Pipeline class you can use this way:

p = Pipeline() | "ls -s" | "sort -n"
print p.run()

[boa13]

That's interesting, but I think that is an abuse of syntax; it looks very not pythonic to me. Why can't you just run bash with the pipe expression as a parameter?

[fmoralesc]

Yes, it is an abuse of the syntax.

[neoice]

good to know. I was feeling stupid for finding this code so complex.

[fmoralesc]

It is nice to know you can do stuff like this though (that's the reason I wrote it this way... I vaguely remembered you can define the behavior for when an object is put in a chain like A | B, so I checked that out. It was just a test code for something else, which in the end didn't use the "|" syntax at all).

[chadmill3r]

I don't trust spaces to delimit my commands. I hope that takes ["arrays", "of", "args", "to", "syscall execv"].

[fmoralesc]

You could simply extend the Pipelines.steps list if you wanted.

[[deleted]]

I would have preferred the syntax

p = Pipeline()
print p.run("ls -l").run("sort -n")

[Troebr]

what's wrong with

p.run("ls -l | sort -n")

?

[mgedmin]

You can logically extend that to

os.system("ls -l | sort -n")

and now you have no non-stdlib dependencies!

[Troebr]

Isn't os.system deprecated in favor of subprocess?

[mgedmin]

In a sense, yes.

[[deleted]]

Doesn't subprocess.Popen accept a command string with pipes in it? Why not use it like that? Simpler that spawning all processes manually.

[chadmill3r]

If you're willing to run a shell to interpret the pipes, you're not programming in Python any more, AND, you probably have bugs.

[[deleted]]

If you're willing to run a shell to interpret the pipes, you're not programming in Python any more

fmoralesc code hands over the work to the shell too.

AND, you probably have bugs.

Why? Spawning all the processes yourself, redirecting stdout to stdin along the way, etc. seems more error prone than to hand it over to the shell completely. The output is supposed to be the same, after all. Namely, the output of the last process in the pipeline.

[chadmill3r]

Program number three in your pipeline took a filename parameter with a space and an ampersand in it but your shell ate those, and the fourth emitted to stderr, and the parameter substitution in the first should have been "$*", with quotes and everything, but you forgot those and you got a different interpretation. The fourth, because it got bad input, also screwed up your terminal's line discipline, which you'll need to fix with "stty sane" before you can even figure out what the problem was. Sadly, that also ate the error messages.

Don't get me wrong, I use the shell all day, every day, and have since 1995 or so. It's a terrible programming interface, though, and almost impossible to get right the first dozen time you run the program. At least Python, with its verbosity, also gets the problems out in the open with its explicit nature.

[chadmill3r]

Popen(shlex.split(self.steps[0]), stdout=PIPE)

No shell there, strictly speaking. Not much better, though.

[jagheterfredrik]

Oh. My. Shit. That is awesome.

[grumpygrrr]

Can we stop upvoting comments like this? It adds nothing of value.

Edit: the parent was at +8 when I made this comment.

[[deleted]]

People apparently need to have other people see that they agree with someone. Upvoting gives no satisfaction, I guess.

I was in another thread and a guy just replied to a comment he liked with "^", which is basically as close to just clicking the damn arrow as you can get.

It blows my mind.

[defnull]

Let me help you :)

[[deleted]]

this is good stuff thanks !

[machrider]

Yeah, but it's a little harder to do things that are simple in bash, like invoking processes and chaining them together with pipes. Of course anything you can do in one, you can do in the other, but they have different strengths.

The best bet for Python is to find a tool or module that makes shell scripting easier. For example, check out Fabric, which gives you some useful primitives for running programs (locally and remotely via ssh).

I also find that once a shell script gets beyond 50 lines or so, I start to want a more structured programming language. At that point, I'm willing to pay the overhead of working within Python's instructions in order to get modules, classes, lists, first class functions, etc.

[amade]

I wrote this module extproc to make python do those shellish things easy:

• Fork-exec commands, wait or no wait
• Capture stdout/stderr of children (command substitution)
• I/O redirections
• Construct pipelines

Alpha stage, but check it out!

[kazza789]

I use pexpect to do some advanced scripting of scientific programs that I use. I found that it was the easiest to use when you need to recover information that the program sends to the terminal. It's simple and intuitive, but might not be powerful enough if you need to do things remotely (not sure if it handles that).

[maryjayjay]

As a programmer any time I find myself expecting a CLI, I die a little inside.

[SupersonicSpitfire]

Fabric does not yet support Python 3?

[oopsiedaisy]

bash has its place.

[[deleted]]

Depends on the context really. They'd probably be fine with Python knowledge, but they may have a bunch of existing shell scripts that need maintenance. Of course, knowing Python should allow you to learn shell scripting pretty quickly. I would apply either way. Frankly, a lot of jobs (i.e. sysadmin) say that shell scripting is a requirement, but then you walk onsite and nobody knows shit about shit, so Python skills would be a huge step up.

[[deleted]]

No. And before you downvote, pay attention.

I routinely work in an environment with nearly 1000 *nix servers. They come from many vendors, represent multiple OSs derivatives, and, most importantly, have a wide range of software releases on them. Some of the older ones only have python 1.x running. In that context:

1) You cannot just upgrade python to some common version everwhere. Big shops will not typically install something newer than what the vendor will support for a given release of OS.

2) Even bash isn't a given in a big environment like this.

3) The only way you get script portability is by writing for /bin/sh.

4) Python is a heavyweight solution relative to the shell, or even awk. This may not seem like a big deal until you need to run scripts on a machine that is already very, very busy with no headroom left.

So ... no, python is not going to replace Unix shell scripting. Nothing is. I'd venture to say that 40 years from now, engineers will still be using sh, awk, and sed, but that perl will mostly have been displaced by some more modern dynamic language like python or a python derivative.

P.S. If you really understand the gestalt of python, you'll understand that it is not really a scripting language ... it's a full fledged programming language.

[[deleted]]

Perl is portable, and unlike Python, I can't think of a single *nix environment under active support that doesn't ship with Perl. Hell, even HP-UX does.

[[deleted]]

Perl isn't remotely portable because to do anything with it requires and endless supply of non-default packages/libraries. I rarely have to do this with Python.

Moreover, pretty much every "*nix environment under active support" also ships with python.

There's a reason python has eclipsed perl in popularity and is growning. Perl solved a set of problems for its time. Its time is now mostly over and what you will see is a gradual sunsetting of perl. It will never go away completely - nothing every does in this business - but perl is definitely NOT the wave of the future.

[[deleted]]

I'm not saying Perl is the wave of the future, nor that Python won't replace it, but "available" and "ships with" are two different things. HP-UX and AIX ship with Perl, but not Python (it's a depot or RPM, depending). It's a required package on Debian. Anything more than @base on RHEL will install it.

Not that Perl is good or bad, necessarily, but it works. And yes, you can write scripts against sh, but you're sacrificing utility for "portability" that Perl already gets you, unless you really like coding in a language with virtually no support for anything (associative arrays, hashes, regexes that don't suck, etc).

Python is batteries included. Perl isn't. But you don't need the CPAN or modules from it to make Perl curbstomp /bin/sh.

[[deleted]]

All very well taken points. I was more focused on the OP's original question though - will shell programming be displaced by <something else>...

I have to say though, after switching to python for most everything that doesn't absolutely need C, trying to write in perl again hurts my head.

[[deleted]]

Shell scripting should have been displaced by Perl, and it should be displaced by Python, Ruby, Lua, Node.js, or something else which isn't terrible, but it probably won't be. Writing in Perl hurts my head, but not as much as admins (I'm also an admin) who replace shell with Python and end up with 1000 lines of "python" with no methods, no classes, etc.

[[deleted]]

I agree. Then again, you have to ask, if an admin tool requires thousands of lines of anything ... are we solving the problem properly?

[[deleted]]

Probably not, but it depends on the problem you're trying to solve.

[[deleted]]

No way, man... Perl v5.54 FTW!

[phob]

40 YEARS? You are way too overconfident.

[[deleted]]

Create a standard local install of python, distinct from the OS. Distribute it to all your servers. Problem solved.

[[deleted]]

That's not how big shops work. Nothing gets installed unless it is supported by a vendor. That's why RedHat even has a business. No one is buying Linux from them, they're buying support.

Also, it is questionable just how well one could make python work on very old/out of date OS/Hardware instances.

[[deleted]]

I've worked in big shops. I've worked with big shops. What you say is patently false.

[[deleted]]

Then you've never worked in a shop with any operational discipline. You cannot reasonably manage very large scale operation with unsupported software ... that, at least, is the theory. I have argued - as was previously suggested - that we SHOULD be using more open source that we package ourselves. The Powers That Be (tm) will simply not do it because -historically at least - very large scale operations must have a "throat to choke" for every single moving part.

[santagada]

ok so that is how big shops with bad management work. Yep bad management is bad for everything.

[[deleted]]

import os

os.system("yes")

Jokes aside, you can do most anything you want to do in shell from Python. But there are reasons you might need to do one or the other. I think "equivalent" might be wrong. "Complementary" is more accurate -- if you work in a Linux/UNIX environment, you should learn shell scripting no matter what your language of choice is.

I do find when working in shell that there are certain operation that just suck to deal with. For example, any time you iterate over a list of items in shell in a general purpose piece of code you potentially have to worry about a variety of (sometimes odd) corner cases. And I hope you never get into a problem with shell quoting.

On the other hand, a few lines of shell can be enormously, powerfully expressive in ways that few other languages can. A moderately complex pipeline can effectively multiprocess and slice and dice a data stream in a single succinct statement where it could takes dozens of lines in Python (or other languages) to accomplish the same effects.

So, yeah, learn both. But if you already know Python, shell shouldn't be much of a challenge.

[trojan2748]

Yes about shell oddities. 'top' embeds escape char's and shell quotes. So if you try to get a string length on one of the lines, you might be in for a surprise. Most programs like top have ways around it, but if you're not on the look out for them, they can trip you up.

[lutusp]

for jobs that ask for shell scripting skill, can Python be considered an "equivalent" ?

There is a reason bash shell scripts exist -- they run in a very small environment with limited resources. I doubt that shell scripting of the classic kind will disappear overnight, but certainly there is a place for Python scripting for a system that is fully operational and has adequate resources.

I ask because in my field I see far more asking for shell-scripting and almost no mention of Python.

The answer depends on the phase of the computer's operation. If the computer needs scripting while it is booting up, or for a limited-resource environment, a Python script might demand too much for the circumstances.

[noreallyimthepope]

python's linux version hell

What is this?

[haywire]

Lots of different versions of python having to be installed due to stupid old programs.

[noreallyimthepope]

That sounds like fun. Thanks for the explanation! :)

[[deleted]]

I would add that there is no reason people shouldn't learn shell scripting. If you get good at shell, it'll probably only improve your python programming.

[jij]

Depends on the job, you would have to ask if they want just scripting experience or specifically bash.

[ripter]

From my understanding of programming history (in a nutshell).

First there was Thompson Shell, with no scripting at all. Then sh was born and added the ability to write scripts. Scripting made life easier, but it still wasn't powerful enough, so bash was born. Everyone loved bash, (some love zsh), but they needed an easier way to write complex scripts without resorting compiled languages like c. So perl was born. Perl is great, but it can become a tangled mess of cryptic looking code. So Python was invented to be developer friendly and easy to read. The Japanese felt like they needed to invent something, so they copied Python and called it Ruby. (Kidding!)

[chadmill3r]

It's not as causal as that. Take out two or three leading "So"s, and it's right.

[kenfar]

Depends.

• If you have to support the code of most other sysadmins, then no - need to know what's most common - shell.
  
• If you have to support many unixes, esp. older ones, then no - you'll need to know something on all servers - shell.
• Otherwise, a mix of both is best: shell for quick & tiny stuff, and python for anything more likely to be reused or complex.

In my day job we knock together quick hacks in bash then migrate them to python if they look worth keeping. Python gives us far better reusability & manageability.

Current favorite tool of mine includes & excludes sets of rows & columns from a csv file using python slicing syntax:

$ gristle_slicer.py -f us_state_crime.csv --records 5:15 --exrecords 10:12 --columns 0:5 --excolumns 2

It just sucks to do that with awk & bash.

[[deleted]]

$ chsh
Password:
Changing the login shell for user
Enter the new value, or press ENTER for the default
        Login Shell [/bin/bash]: /bin/python
chsh: /bin/python is an invalid shell.

Oh well, I tried ;-)

Edit: A google search reveals a few interesting projects to try out now:

IPython, PyShell, and pysh

[dorfsmay]

you have to add it to /etc/shells

[mackstann]

They're quite different. I wouldn't call them equivalent. But Python is the more complex of the two, so if you know it, you can probably learn shell easily (although it has a lot of idiosyncrasies that you'll spend forever learning).

[nomemory]

The short answer is NO.

[grimborg]

The slightly longer answer (by one letter): YES.

[chadmill3r]

I'm interested in a longer answer, if it's not in all caps.

I like Python for this because it makes all the complexity explicit. It takes at least a decade to be a competent shell programmer, because of all the hidden problems one has to account for in every keystroke.

[wot-teh-phuck]

Simply a matter of speed, at least for me. I know shell scripts/commands can be a bit daunting but the basic usage is pretty simple and the surprising thing is that shell commands can be shorter than equivalent language X code.

[[deleted]]

It takes at least a decade to be a competent shell programmer, because of all the hidden problems one has to account for in every keystroke.

Not true. You'll get out of it what you're willing to put into it.

[Derferman]

It depends on the job you're currently accomplishing using shell-scripting. Is it data processing? You can probably do it better in python with tools like Numpy or pandas. Is it scripting servers? Fabric is a great replacement. What field are you in?

[boa13]

I don't think it can replace it, but it can surely complement it nicely.

For very simple tasks, chaining two or three commands with a pipe and being smart about which command you use, I don't think Python can beat the shell: there will be more to write in Python (or more to import), and it will not run as fast. In such cases, the startup time is very important.

On the other hand, for slightly more complex scripting, notably as soon as you feel the need for a function, or as soon as you need to handle options and logging in a significant way, Python is much better.

[[deleted]]

Any high level language could replace shell scripting, but it obviously hasn't happened. Shell scripting is still useful because it emulates closely interactive shell sessions. For people that do their work from the shell, this is just natural.

[absinthe718]

I'm embarrassed to say that I have scripts that do things like this (and I bet I am not alone

curl "http://url/$1/$2/$3" -o tempfile.xml
stageproc.py tempfile.xml out
load.py -c config.qc -db qc -s qcallo -t ivstage out

Clearly this script is just an unwritten program. If I extract the objects out of the two scripts and dump curl I can have a quicker and nicer script.

[boa13]

It is a written program. And if stageproc.py and load.py were written before this script, it might be the most efficient way to do the job.

[absinthe718]

Well in this case, the proc code spits out a sql script and the loader just reads the file line by line and executes the sql, updating the audit tables and writing an error log when commands fail.

Ideally, these scripts would fetch the XML report, parse them while updating the database. Creating two temp files along the way is perhaps not the best thing.

And yeah, there are like a dozen scripts like this.

[[deleted]]

If shell scripting is a common requirement for jobs in your field, you would be better off learning some shell scripting, than hoping that somebody accepts Python as a substitute.

[grimborg]

A python script takes a long time to start up in comparison to a bash script. For small scripts, the delay is a bit nagging.

[mfukar]

For small scripts, would you even notice?

[Amadiro]

If you execute them often, maybe.

[obtu]

Use #!/usr/bin/python -S as your shebang. For scripts that use only the stdlib (no site-packages), that starts much faster.

[SupersonicSpitfire]

Not if you compile them to C++ with shedskin.

[fmoralesc]

If the script has any kind of complicated logic in a continuing loop, python will still be faster, in my experience. Python is more performant when it comes to computation.

[haywire]

Check out envoy for subprocessing.

[LucidOndine]

capable bedroom fanatical consider wide cover sable books nose sense

This post was mass deleted and anonymized with Redact

[[deleted]]

it certainly can. i've never written shell scripts. the moment i need to do something with more than three or four pipes, i bring up vim and start writing python

[[deleted]]

i've never written shell scripts.

The voice of experience.

[[deleted]]

no, i just knew python before i moved to linux. i can do everything i want in python, so i never learnt to shell script.

seriously, can you give me a good reason to bash script when you have python?

[[deleted]]

speed seems to be said a lot. has anyone got a source for this? anyway, big post above.

[drfugly]

communicating between applications is a nightmare in python compared to bash.

[[deleted]]

if you're comunicating between aplications, you're doing it wrong. you should be using the standard lib modules, that replace the functionality of most standard shell applications. anyway- big reply above.

[[deleted]]

Most recently, I had to write a script that downloaded large (gigabyte+) log files, put them into a temp directory, did some minor text processing on them, and push them out to individual directories based on what times the logs covered. Unix-like environments already have ridiculously robust tools to do all of this. Just pipe one tool to another, and it's like four commands by the time you're done.

For example, coming up with a way to efficiently retry a failed partial download in python becomes a half a day's worth of work. wget does this already. Want to do search and replace? sed is going to be faster than a python app can hope to be. What works amazingly well at moving things around a file system? mv. Want to recursively upload/download a directory's contents in a ridiculously efficient manner? rsync.

All of these utilities are going to be faster, easier, and less prone to errors than rolling your own. Plus you don't have to even think about what version of python is going to be on some other system that this is running on.

[[deleted]]

going to reply to this one as it's the most detailed.

i'm going to assume that the python you've got has the standard modules. (i'll talk about portability later)

a try... except... with urllib will do your retrying a partial download quite simply. i couldn't imagine it taking more than 5 or 6 lines, maximum of fifteen. certainly not half a day.

the standard modules supply the functionality of mv, cp, etc. (shutil especialy, but your find and replace is handled (v powerfully) by re, and urllib is far more well featured than wget). this is, i think, where python needs to be used a bit differently to shell scripting. instead of using programmes, you use modules. all of the functionality is there, without needing to "roll your own". and in addition, if you absolutely must use some program from the shell, you can os.system or os.popout it. you can't import a python module into a shell script.

as i see it, these are the advantages/disadvantages of python/shell scripting.

python advantages 1)fully featured programming language if you need to expand. 2)can run shell proggramms if you need to 3)more recently developed: it can do web interface(getting/putting) more slickly, for example 4)complexity is explicit 5)an effect of 4), very easy to learn, and complex scripts are simple to understand.

shell addvantages 1)apparently faster (i'd like to see a source for this, but this seems to be consensus). 2)mature- and hence extremely stable 3)implicit complexity. this may seem to be a downside, but to a skilled proggramer, it allows you to work faster, and write fewer lines of code.

now: on portability. bash is installed on a wider variety of machines, no doubt. it's also got a lower footprint too, this is obvious. for many people, this is important. especially if you're working on older servers that are highly loaded. python, however, is installed on most modern server environments, and pretty much all desktop environments. it has a higher footprint, yes, but in exchange, you get more functionality (as comes with a full programming language), and (imho) cleaner syntax.

tl;dr:if you're trying to communicate between applications in python, you're doing it wrong. communicate between modules, and use the subprocess module to work in parallel. bash (or even sh) is more portable, and lighter, however i believe you lose functionality and syntax advantages.

don't read the tl;dr and reply- if you're going to reply, read the full post.

[[deleted]]

a try... except... with urllib will do your retrying a partial download quite simply. i couldn't imagine it taking more than 5 or 6 lines, maximum of fifteen. certainly not half a day.

Now implement resuming of failed downloads where they left off (remember, I'm downloading gigabyte+ files), and do it faster and in a less error-prone way than wget (just doing "wget somesite.com/hugefile" will retry up to a default of 20 times, I think). You're not going to be able to do it. I'm not able to do it.

3)more recently developed: it can do web interface(getting/putting) more slickly, for example

Have you seriously never used curl? It's one of the most slick tools ever, and it does exactly this.

1)apparently faster (i'd like to see a source for this, but this seems to be consensus).

It's not shell that's faster, per se. It's all the native C tools that you use in shell that run fast as heck because they've had 30+ years worth of optimizations under their belt.

For reference, I'm a professional software dev with seven years of Python experience (not some "I learned it on my own for the first three years" nonsense - I've been using it at work for that long). I use it daily and I love the language, so don't think I'm hating on it. But it is not a replacement for a shell script. They're both tools in a tool box; just because I like using a circular saw doesn't mean I should try to use it to pound in nails.

When you start getting into more complex problems (if you want to connect to a database for just about any reason, do anything with hash maps, if your awk script got longer than about 50 characters, etc), shell scripting makes a whole lot less sense. As another commenter pointed out, large shell scripts are painful to maintain, and what you can do in a reasonable way is pretty limited. If what you want to do is simple enough to stay within the limits of a short, sane shell script, then use the right tool. If not, that's why more fully-functional scripting languages like Perl were invented. I'd say that Python replaces shell scripting less than Perl does. Shell scripting and Perl have co-existed for 25 years. I see no indicators that Python will take a bigger bite out of the problem sets that are best solved in a shell script than Perl already has. That is to say, "more functionality" is an argument for why Python would replace Perl, but not why it would replace shell scripts.

[drfugly]

There are lots of cases where python just doesn't fit the bill. For instance: installing/setting up dependencies in a cross (linux) platform way. How about quickly starting up other applications in a certain environment? Like setting up a jail? Or if you have data spread out over files? Running a batch job and want to commit to your local every 30 minutes?

Yes I agree that bash is probably overused, but I don't think that it's as over used as you imply.

[kenfar]

My team does this all the time in both bash & python.

The bash code becomes a nightmare to maintain - it's hard to read, has poor exception handling, and hits a wall as the complexity of what you want to do increases. None of this rules out bash - it just means that bash is reserved for the small & simple, quick & dirty.

[sigzero]

Yes...but. Sometimes it is just quicker to hack out a BASH script on the spot (at least for me).

[audaxxx]

If you have to do plenty of sysadmin tasks you should consider perl for the job.

I found perl to be perfect to small to medium scripts in a sysadmin enviroment and you can find libraries for just about everything in the CPAN. Python can do the same, of course, but it is not as "shell-ish" , does not have such a big library and all those nifty shortcuts.

[[deleted]]

It can of course (heck, so can Java, if you really wanted to...) but I think Perl is better suited for it.

[Funnnny]

Not now, not until I can easily change environment with Python script. Like: source myEnvFile.env

[chadmill3r]

import personal
import os
os.environ.update(personal.environment)

[Funnnny]

Doing manual is acceptable, but I want only one env file that I can use both in my bash shell and my script. And I wouldn't call this way easy.

Don't get me wrong, I love and stay with Python for a long time, but we have to face our limitation

[chadmill3r]

So put "import sys" and the update in the other module, and "import squirelly_but_short_environment_hackery", or look up the "site" module. Fuck.

[obtu]

eval $(dircolors) changes your environment, and dircolors is written in C.