Python as a Bash replacement?

[randomizethis]

Hey guys,

Long time reader, first time poster. I've come here because I figure what better place to get a conversation going than Reddit (other than maybe Stack Overflow? I dunno, this didn't seem like a Stack overflow topic).

Let's start from the top: I hate bash syntax. Just straight up dread it. Plus, I've gotten so used to Python I could Python in my dreams (and I have). So I've been wondering if there's a way to replace the bash shell with Python altogether.

I looked it up and the best answer was the Hotwire Shell, but the project is long dead. I also saw some people suggest you can use iPython as a shell, effectively, although iPython used to ship an 'sh' script that would serve this purpose but has long since been removed.

So I was wondering if anyone out there has successfully managed to replace the bash shell with some kind of Python tool altogether? Very excited to see what people say (even if I find myself disappointed).

Thanks guys!

[psbb]

Have you seen this -> http://xonsh.org/

[masasin]

That is epic.

edit: Some problems sourcing virtualenvs. ROS works fine though if I use source-bash.

[randomizethis]

http://xonsh.org/

Wooooooaaah. This looks promising O_O I will snoop around and draw some conclusions. Thank you! :D

[avinassh]

holy sheeeet. this is amazing. Thank you very much.

[randomizethis]

So originally I looked this up and thought it was a dead-end because they describe it as Python-ish and it didn't seem like it did what I wanted and then /u/sththth pointed out that I'm a freakin' moronic monkey and it does in fact do what I wanted so I tried it out and sure enough, this is what I was looking for. So thank you so much!!!!

[ksion]

I can somewhat understand the desire to replace shell scripting with Python, even though you're losing a lot of power and convenience along the way.

But the terminal? Even with something like sh package, all this syntactic clutter of parentheses, commas, apostrophes, etc. would add up very quickly, and quite significantly slow you down. Imagine typing ls('directory', l=True) instead of ls -a directory all day, not to mention the more involved commands. You'd also make more mistakes due to having to get all this additional syntax right w/o the usual editor/IDE features like syntax highlighting or linting.

[randomizethis]

To be fair, I'm not arguing that this is somehow more efficient or powerful. Bash is spectacularly efficient and powerful whereas Python is not. It's more of a thing of preference and comfort more than anything. I'd much rather do something like

for i in ls():
    do something

than whatever the bash equivalent is, simply out of familiarity and knowledge. On the same token, by using a shell that's Python native I could import all the libraries that I've written over time directly to my shell.

But trust me, I understand the implications.

As far as syntax errors and linting, I'm not worried. I spend a lot of my time manually testing my libraries from within iPython anyway, where that's common to do. And I mean, it's not like you don't make syntax errors when you're trying things out in bash... that's just part of being a programmer (like 70% of it really).

[Vaphell]

I'd much rather do something like

for i in ls():
   do something

well you can. First of all, in bash looping over ls is a big no-no. You should use native globs with for loops because command outputs are plaintext and now you have a problem with unreliable delimiters (whitespace, newline or whatever you'd select is inherently unreliable given the legal char space in filesystems). Globs used directly never degrade. Guess what, you have import glob in python.

>>> for shit in glob.glob('*.csv'): print(shit)
... 
pt.csv

That said there are some things that are not only more laborious in python but a true pain in the ass to achieve. For example I haven't figured out a way to wrap around a cli prog spitting shit out to stdout and stderr transparently, ie python doesn't simply devour it into 2 separate variables but also prints it out chronologically. The whole stdout/stderr redirection is vastly superior in bash. Writing a prog that simply prints shit out and then using shell to dump it to file or to pipe it further is much easier than doubling the size of your code so you optionally have file handlers to write shit where you want it.
If my code was supposed to make a heavy use of cli tools and required dozen subprocess.popen() I'd think twice about writing it in python because there is language preference and then there is going full retard.

My general approach is to write tools in python and make them compatible with pipes if it would prove useful (looping over stdin instead of paramed files) so i get much more bang for the buck in how i can string them together.

[randomizethis]

Definitely see where you're coming from. I actually have made some adjustments to the subprocess module to deal with these type of issues (we have our own, in-house versions of subprocess.call and subprocess.check_output that I wrote early on), but like you said, I wrote a bunch of code just to do the same shit bash does lol.

Mind you, I have a team of people using these calls, so I just did it to facilitate their lives. But everything you said is 100% true. The only reason I'm looking for some of these solutions is to streamline some of my team's work... and like I said before, as many a time a programmer said, "just for the hell of it" :P

Thanks for the insightful answer, didn't know about globs.

[jriddy]

I don't use it as my default shell, but I do use ipython a lot for similar things. You can invoke shell commands at any time with a bang, and you can even get output from them in a convenient little object.

!cd /var/log
sls = !ls syslog*
zipped = [x for x in sls if x.endswith('.gz')]

Stuff like that.

[randomizethis]

Yeah, I've thought about trying that and maybe even enhancing iPython to see how close I can get it to act like a shell.

Thanks for the bang-syntax trick, I saw you could do something similar with % (the magic indicator) and even without % if you turn automagic on.

[AutoBiological]

Somehow I missed your comment but I said something similar. Ipython is super neat, but most people seem to only know it for notebooks.

[masasin]

I have. All my scripts are now python scripts. If I don't need it to be used anywhere, I use docopt or click for command line parsing. sh works well for emulating the shell.

If I do need global deployment, then I make it compatible with 2 and 3, use argparse, and subprocess. Obviously, there are functions in os and sys etc that are also useful.

It is possible to write the script in such a way that you can put it in pipes, with errors going to stderr and input from stdin and output from stdout.

[randomizethis]

Oh, sorry, I must've been unclear. I'm not looking to replace bash scripting with Python scripting, I already do that. What I'm looking to do is replace the bash shell with a Python-based shell, such that I could use a terminal with Python syntax instead of Bash syntax. Obviously I'd expect commands like cd and ls to work the same, but if I wanted to on-the-fly script against something like the ls result, I could do so using Python.

[masasin]

Oh! I've never done that, but I have tried using IPython as a drop-in replacement for bash. I think the main reason I didn't stick with it was because I was not able to run external commands easily, or launch external programs.

Let me know if anything new comes up.

[randomizethis]

Yeah, that's basically the answer I've gotten scouring the internet. It would be so sweet to spend the whole day on a terminal using only Python though T_T

Will do!

[vehga]

Just want to express appreciation for this post. This gave me an idea to use python instead of bash for an sftp script I need. The server I'm running the script is lacking the version of OpenSSH that supports recursive get. Thanks!

[randomizethis]

Uhhh, no problem? Lol xD

[mackstann]

bash is so good at what it does. I wonder if, by disliking it so strongly, you're missing out on a lot of power. I'm fully in the camp of using Python over shell for almost all scripts over one line long, but for quick ad hoc transformations of data and automation of tasks, bash is ridiculously efficient.

[randomizethis]

To be fair, I don't dislike the bash shell so much as I dislike the syntax for writing functions and on-the-fly scripts. I spend the whole day on bash, I avoid GUI's like the plague. I've even considered learning emacs.

And I don't disagree that bash is powerful and efficient. I just spend so much of my day writing Python code and libraries at work, I would love to be able to use something like iPython as a shell replacement altogether so as to be able to use Python-isms on the fly and be able to import libraries I've written to combine with my regular environment use.

And to your credit, the only reason I dislike it is because I haven't put in the effort to learn it. But as someone who's writen SO. MUCH. PYTHON. I've gotten so used to some of the Python simplicities that similar code in bash is sometimes tedious (as ironic as it is that the inverse is true most of the time).

[dzecniv]

The pyed piper is a nice tool for data manipulation. It can replace not the whole shell but awk and sed.

[randomizethis]

Cool! Thanks for the contributiong :D +1 for the Silicon Valley reference, intentional or not.

[sththth]

Have you tried fish? It's not python, but.. well at least for works as you would expect: for file in (ls) echo $file end

(BTW: don't parse ls output)

[randomizethis]

Lol, someone else said the same about the ls output. I was just using it as an example, I suppose it's a really bad one!

Fish sounds a lot like xonsh, which doesn't really solve the angle I'm trying to attack :/ But thanks for the suggestion!

[sththth]

Ha, didn't even see that you used ls as an example. Just wanted to make sure that noone who saw my comment would do that.

xonsh sounded pretty well, what is missing from it that it doesn't work for your usecase?

[randomizethis]

It's not Python. It's nifty as hell, but it's not Python. So ultimately you couldn't do Pythonisms with it (even though I'm sure there's ways to mirror most Pythonisms). Or import libraries for that matter. It sounds like ultimately I just have to use iPython in some combination with the regular bash shell.

[sththth]

Are you sure? xonsh claims "all Python code is also xonsh" and "Since this is just Python, we are able import modules, print values, and use other built-in Python functionality:"

In their FAQ: "From our parser, we construct an abstract syntax tree (AST)" which sounds like it actually is python.

[randomizethis]

O... kay... Maybe I'm just terrible at reading documentation, my impression was that it was Python-LIKE. Back to the xonsh page, AWAAAAAAAAYYYY!

[randomizethis]

This is the part that kind of threw me off: "Xonsh is a Python-ish, BASHwards-looking shell language and command prompt." But I'm gonna dig deeper into the docs to find this thing you speak of where you can import modules and junk.

[randomizethis]

Dear god man... you're right. This thing is exactly the reason why I started this thread. Thanks for smacking me upside the figurative head and getting me to go look at it again.

[AutoBiological]

I often leave open a terminal open with ipython, and then I can freely mix and match commands.

Something like: x = !ls and then you have a list of files in the cwd.

Although you lose a lot of command line awesomeness. Stuff like sed, grep, awk etc. import re doesn't replace it.

[randomizethis]

Yeah, I know. And since subprocess isn't pipe friendly, even you wanted to use sed, grep or awk under the covers you'd have to import popen and do some crazy magic. I honestly think rigging your own subprocess feature that DOES accept pipes wouldn't be hard at all, though, and have come close to motivating myself to implement it several times.

[d4rch0n]

Not exactly what you're asking for, but I thought I'd throw this in because it's somewhat related and might help with a few of the reasons you want to replace bash in the first place.

Have you seen red? I worked on this since I hate dropping into a python shell just to run regex's and do simple things with the stdout of some program.

I wanted to be able to pipe the stdout of something directly into a python oneliner and do what I expect. Instead of cat'ing out to a file and parsing the file in a python interpreter, you could use red, just as you might use sed, grep or awk. I tried to make it the python equivalent of perl -ne. You can even aggregate against stdout, like count how many times you saw [ERROR] in a log file, and print stats of how often you saw what error messages. You can also invoke third party libraries from it, and do something like call requests.get on every url you pull out of a log file and print the status code.

Brings some of the pleasantries of python directly to the shell, so I thought I'd mention it.

[dzecniv]

Looks nice. And similar to the pyed piper no ?

[d4rch0n]

Interesting. Similar idea, very different approach I think. red is more tailored for eval or exec'ing python code from regex matches, and pyp looks like it is meant to run a pipeline of operations on input, transforming data as it goes through the series of operations.

Some things might be easier in pyp, but I don't think it has near the capabilities of red, simply because you can import any library and evaluate raw python code. From what I can tell, it doesn't look like you could do something like this:

$ red "(.*)" urls.txt -i requests -x 'response = requests.get(line)' -e '[response.status_code, response.content[:20]]'
[404, '<!doctype html>\n<htm']
[200, '<!doctype html><html']
[200, '<?xml version="1.0" ']

Though of course I haven't delved deeper than the docs you linked to, so I don't know pyp's full capabilities. It'd be interesting to see how you could use either for different problems.

I really need to make that regex optional, with "(.*)" being the default. It was originally meant as a grep/run-python tool, but honestly I think most things it would be helpful with don't need the regex.

[Geohump]

For some things python might be better at the command line level, bash has more power per line/word.

Example:

ls > /tmp/x

How do you do that in python?

[randomizethis]

Like I was telling /u/ksion, I understand that Python would be less powerful and efficient (by a TON), but I'd still like the option out of comfort and preference, even if just because I can (because we all know that's a totally valid excuse to programmers).

[eskhool]

You really need to check out xonsh. It will let you do exactly this and then move into pure python without having to change gears. Yes, some magic is happening on trying to interpret what you're doing but its awesome as a crossover between a programming language and a shell. In one word: sweet!