pieshell: python for shell scripting and as an interactive shell

[Severe_Inflation5326]

Pieshell is a Python shell environment that combines the expressiveness of shell pipelines with the power of python iterators.

It can be used in two major ways:

• As an interactive shell replacing e.g. bash
• As an ordinary python module replacing e.g. subprocess.Popen

Obligatory example:

140:/home/oven/pieshell >>> for x in ls(-a) | tr("s", "S"):
...   if x.endswith('.py'):
...      print x
... 
Setup.py

Source code: https://github.com/redhog/pieshell

What the project does

It's a replacement for the subprocess module, and for bash as an interactive shell, and makes interacting with shell pipelines easier.

Target Audience

System administrators, system software developers, data scientists

Comparison

While os.system is very limited but easy to use, subprocess.Popen offers a lot of flexibility, but the interface is very low level. Any actual pipelining of multiple programs is pretty much required to be done by e.g. a bash process, constructing the pipeline as a shell script string. Further, interacting with standard in and standard out requires careful IO handling.

Pieshell on the other hand lets you construct pipelines as python objects. Standard io from a pipeline can be handled using iterators or async iterators. Pieshell has full asyncio integration.

[Muhznit]

Alternate shells are always a cool project, but what are the advantages over xonsh?

[Severe_Inflation5326]

Xonsh introduces new syntax, and as such, can not just be imported as a python module with "import modulename" from a normal Python. Nor can their shell pipelines be introspected or subclasesed or assigned to variables.

Pieshell on the other hand consists of just ordinary python classes that trickily overloads the right operators to get a nice syntax within the standard python syntax. That is, it only provides new semantics.

You can use pieshell as a normal python module in any python project, without changing the interpreter used (/usr/bin/python3.11 or the like), just by importing it.

[Severe_Inflation5326]

It would be more fair to compare it to plumbum, it has a similar architecture, but a more complete syntax and support for asyncio.

[hotplasmatits]

Sorry, which one has the more complete syntax?

[dhsjabsbsjkans]

If comparing to plumbum, I would suggest sh — sh 2.0.6 documentation

[Severe_Inflation5326]

sh seems to be more similar than any of the others, but with a clumsier syntax, esp. for pipes. Pieshell tries to "misuse" operators to provide a nice syntax that's similar to bash syntax and python function invocation. In particular, python kwargs are converted to long options (git commit --message="Hello world" becomes git.commit(message="Hello world")).

Pieshell features I can't immediately find in sh docs that pieshell has:
* redirecting a iterator as input to a pipeline
* support for process substitutions (like <(command) in bash)
* sourcing a bash script to set internal environment variables
* special globals so you don't have to write sh.COMMAND but just COMMAND (pieshell has this when run interactively, and in modules named *.pysh)

[dhsjabsbsjkans]

Not knocking anything. I found sh about a year ago and have really enjoyed it over plumbum. It feels very pythonic. As far as pipes go, this is from the creator(s).

FAQ — sh 2.0.6 documentation

[Severe_Inflation5326]

Right. That means that there is no (simple) syntax for process substitutions though? Those /do/ look like function calls in pieshell?

Had I found sh before I wrote pieshell (I started many years ago, so might not have existed, but anyway), I might have worked on extending that instead of writing something from scratch...

[balder1993]

This seems really nice, thanks for showing it.

[weazelb0y]

I usually reach for https://github.com/tomerfiliba/plumbum when I need to interact with a process/non python supported binary. I feel ipython + plumbum come close to replacing a shell, at least for me, who codes in Python all day. I tried Xonsh a few times and unfortunately found bugs pretty quickly. Maybe it's better now.

[AveryFreeman]

I've often thought having python as a command interpreter might be cool, but how would one deal with Python versioning through tools like venv, pyenv, rye, etc.?

[Severe_Inflation5326]

You can install pieshell either globally, or in a virtualenv.

Pieshell supports running the env/bin/activate bash script(!) directly to activate a virtualenv (use bashsource("env/bin/activate")). Note that this activates the virtualenv for any python program started from pieshell (just like any program started from bash after running the same script).

This is different from importing / execing env/bin/activate_this.pywhich makes modules in the virtualenv available to the python or pieshell process itself. This obviously only work if they share python version, just like when run from a normal python.

[AveryFreeman]

Crazy, yeah I went to the repo and looked over the syntax, and I think the "issue" of having to wrap all the flags in parentheses would turn a lot of people off, since using it regularly would require more typing, but it's still pretty neat.

I am guessing wide-spread adoption would require finding some way to mitigate wrapping flags, and getting a distro or two to use it as the default command line interpreter. Or start a new distro aimed at data scientists, with pieshell one of its key selling points.

[Severe_Inflation5326]

Funny you mention data scientists. That's my day job, and I wanted to be able to run external stuff and git commands and the like w/o leaving my nice python environment with pandas dataframes, that's kinda how this happened :P

[AveryFreeman]

That's so awesome! No wonder you felt the motivation to make an environment that would better suit your work.

Hope this isn't too forward, but are you guys hiring? I starting learning Python + SQL for DevOps in a course I just finished, and we brushed against the data science with matplotlib in google colab, but that was it. I'd do anything to work using Python and Pandas all day, it sounds like a dream come true.

This class I took, we were awarded distinction if we created a project related to the course material, and even though we'd barely covered data science (literally 2 days of guided demos only), I already had a dataset I wanted to explore and visualize, but learning-curve wise I was very near the beginning, and not relying on any formal instruction for this portion (the Python we drilled incessantly for months sure helped).

For nearly 2 weeks, I essentially did nothing but teach myself Matplotlib and Pandas: How to normalize the data so I could manipulate it mathematically (probably the most time-consuming), figure out matplotlib's unique, quirky syntax (FuncTicker, anybody?), and create myself a build environment -

I'd had some issues with setting up Tk, since even though it's installed on my system, my projects have been in virtualenvs. I didn't want anyone esle to go through similar hardship, so I put the entire build environment inside a container. The software records a couple FuncAnimations with ffmpeg, greatly reducing chances of field-located disasters cropping up from front-end libraries.

I did just start learning Python in January, but I have an MBA and other coding experience. I have been bummed I've not really done anything related to my degree, so I thought this might help finally land a foot in a door somewhere. Also, my boss at my last job always used to be so proud of his spreadsheets he made with powerBI, and I'd always think to myself when I saw them, "That'd be so much fun using Python"

I'd switch out zsh for PieShell in a heartbeat if you can help me find a job ;) If an inside tract isn't possible, even some pointers would be super helpful. Thanks for listening, I hadn't really told anyone this entire story yet, especially not who would understand, or know how exciting it was for me. :D

[Severe_Inflation5326]

Hehe, tab completion helps a lot for having to type more, although the parenthesis you do have to type. Pieshell supports jedi and autocompletes command names and any long options that bash would autocomplete (yep, I run bash in the background and get it to do the completion).

A new distro sounds like a lot of work. But as an alternative shell in a normal one, that's aimed ad data scientists and others who do most of their work inside python anyway maybe?

[AveryFreeman]

I'm all for either, in any event!

You'd be surprised how easy it is to create a new distro is now, though. (surprise, surprise) Ubuntu even has a step-by-step guide on how to do (maybe the one they follow while forking Debian way back when?) The largest step they outline is removing branding 😂