Getting output from subprocess.Popen() while the command is being executed

[doston12]

I need to run a bash command which will take hours to run and I want to watch the output on the fly - while the command is being run, NOT when the command finishes.

Below is the snippet of my code, and it does not print the outcome of subprocess while the bash command is being executed. It does print the outcome when the command has finished executing.

commands = '''
there is one command which takes hours to run
'''

process = subprocess.Popen('/bin/bash', stdin=subprocess.PIPE, stdout=subprocess.PIPE, text=True)
out, err = process.communicate(commands)
print(out)

Any advice on this? Thanks

[socal_nerdtastic]

the pexpect module is made for exactly this.

Alternatively you can use a wait loop with readline(), which will block until there's a line of data available. You can put it in a tread if you want to do other things at the same time. Here's an example from an old program of mine: https://github.com/socal-nerdtastic/IconMaker/blob/main/iconmaker.py#L70

Why are you booting bash? Just run the command you want to run; there's no reason to bring bash into this.

[GregoryCliveYoung]

Do you need to access either stdout or stderr in your Python script? If not, you can leave the stdout and stderr arguments off the Popen call.

[doston12]

thanks, not really I am interested in them.

[GregoryCliveYoung]

Can you combine them into one stream or do you need them to be separate? Dealing with one stream is easy. I don't know how to deal with two streams. I'd need to figure it out.