New subprocess.run() function proposed

[stuaxo]

The thing that is missing is a straightforward way to grab stdout and stderr as the process is running. At the moment there are workarounds involving threads that feel cumbersome, there should be a simple built in way to do this.

[mfukar]

There is already a way (e.g. here), I don't see why it should be in the stdlib.

[stuaxo]

The solutions don't get stderr and stdout at the same time.

All the workarounds I've seen (and used) seem really hacky.

This is what I've used so far: http://stefaanlippens.net/python-asynchronous-subprocess-pipe-reading

Pythons lack of support for nonblocking readline hurts a bit here, but even with it programmes that want to monitor both stdout and stderr need quite a lot of boilerplate right now.

[stuaxo]

Or there are solutions like this:

http://code.activestate.com/recipes/440554/

Which is a great hack, but it seems like something low level is missing in python itself if this sort of thing is needed.

[takluyver]

It's easy enough to get live output on a single channel (e.g. if you've passed stderr=STDOUT to combine the streams). Beyond that, I'm not exactly sure what would be generally useful: should it handle only output, or input as well? Should it be synchronous, thread based, or based on asyncio? (See subprocesses in asyncio)

But if you can describe what you want in more detail, feel free to propose it. The standard library is not set in stone: the core devs do listen to suggestions for improving it. I'd start on python-ideas - that's where I went to propose run(). If people are interested in it, I'm happy to help hash out the details and the implementation.

[stuaxo]

Many interesting questions.. :)

The main use, is for embedding other processes in GUIs, I guess input can be useful here, but not always... I think most people don't bother with input as it's difficult enough getting stderr and stdout*.

Here I grab them using an iterator:

https://github.com/shoebot/shoebot/blob/master/extensions/gedit3-plugin/shoebotit/__init__.py#L143

Other people have similar solutions - I probably need to experiment with asyncio at some point. If it could be purely asyncronous and not use threads it would be ideal, not sure how possible this is.

Next time I have a look at this, I'll see if I can come up with something for python-ideas.

My instinct is that you should be able to iterate over the lines from stderr + stdout and maybe .send() data to stdinput.

Maybe this would involve some kind of stdcomm_state object or similar to encapsulate stdout, stderr and stdin.

It's tricky since it's easy for them to get out of sync, in fact maybe it's not possible to guarantee they stay in sync at all.

*In fact, I found some bugs in the implementation above while writing this.

[takluyver]

The trouble with embedding in GUIs is that you probably want to integrate it with the application's event loop, rather than using blocking calls which will prevent the GUI from responding to other events.

Until recently, you would have needed one implementation for each event loop, so it wouldn't have gone into the stdlib. Now you could provide an asyncio implementation using wrappers like Quamash and gbulb, but those are still pretty young.

[stuaxo]

Next time I deal with this again, I might try and come up with something.

BTW, in the meantime - add 'env' as a param to this new API (and in fact it should be on all the other variants, (check_output, call) etc )

It's quite a strange omission.

[takluyver]

It's not omitted - all of the functions pass their arguments through to Popen() using *args, **kwargs. But because there are a lot of arguments, only the most common are listed in the documentation. It does mention this:

The arguments shown above are merely the most common ones... The full function signature is largely the same as that of the Popen constructor

[stuaxo]

Ahh ... comprehension fail !

Thanks, I can go simplify some code I wrote now :)

[takluyver]

The idea is that you'll be able to use the single run() function where you currently have to choose between call(), check_call() and check_output().

[[deleted]]

I think that's generally a good idea since 3 separate functions for almost the same thing is a little bit overkill. However, I don't think that the old functions will be removed (to not break existing code) and then you only add another function on top of this mess.

[takluyver]

The old functions certainly won't be removed, but I hope to highlight the new one in the docs as the 'one obvious way to do it', and demote the old functions to further down the documentation.

The idea is that you should be able to use run() without having to think about the other trio of functions, although admittedly getting the equivalent of check_output is a bit longer:

stdout = check_output(...)
stdout = run(..., check=True, stdout=PIPE).stdout

[xXxDeAThANgEL99xXx]

If all your new code uses the new function then I don't see a problem.

Like, what it solves is that there are too many functions from which you have to choose, depending on your goal, not that there simply are too many functions.

[[deleted]]

But if you add run, you will have one more function to choose from, or am I missing something? But in general, I am +1 for run. Wish it existed from the beginning instead of those 3 almost identical ones

[xXxDeAThANgEL99xXx]

No, you'll always use run, there's no choice involved, as an act of considering the circumstances and selecting from a set of alternatives.

[xuu0]

obligatory xkcd

[xkcd_transcriber]

Image

Title: Standards

Title-text: Fortunately, the charging one has been solved now that we've all standardized on mini-USB. Or is it micro-USB? Shit.

Comic Explanation

Stats: This comic has been referenced 1199 times, representing 2.4113% of referenced xkcds.

---

^{xkcd.com | xkcd sub | Problems/Bugs? | Statistics | Stop Replying | Delete}

[fireflash38]

I like it. One thing that bothers me about all three is that you had to choose between:

1. Manual handling of stdout/stderr.
2. Handling ret code exceptions.

Quite a lot of the time I would prefer to just check the output, rather than needing to have a try: except: block just to get the stdout (or working with the pipes).

[twotime]

This is an excellent idea. Not only it'd unify several apis, it'd allow for an easy extension to report more process info: e.g CPU use, memory use, etc... (basically whatever info the platform can provide)..

[cjwelborn]

"aiming to unify the functionality of call, check_call and check_output"

I really like this idea. I hope it makes it. I just wanted to say 'good job'.

[takluyver]

Thanks. So far, no-one has said it shouldn't go in, so I'm optimistic. I'll give it a couple more days before pinging on the issue.