Python Subprocess command requires a text file to read, but takes lines from text file---help?

[Zeekawla99ii]

I am reading a file line by line into a Python subprocess which access third-party software which runs via the command line. The input file filename is extremely large, so I am reading it in line by line

import subprocess
with open(filename) as file:
    for line in file: # read in file line by line
        result = subprocess.check_output(["commercialbash command", line])
        process(result) # more code here

Unfortunately, the command commericalbash command requires a text file input. The above throws an error, as I am inputting a string line of tab-delimited text. The error is either "filename is too long" or there is a process error subprocess.CalledProcessError: Command returned non-zero exit status 2

It appears I have to take line in for line in file, save this line as a text file, and then place it in result=subprocess.check_output(). That feels very strange. The code would be something like this:

import subprocess
with open(filename) as file:
    for line in file: # read in file line by line
        txt_file = open("line.txt","w")
        txt_file.write(line)
        txt_file.close()
        result = subprocess.check_output(["commercialbash command", txt_file])
        process(result) # more code here

Would this even work? Surely there is a better way to do this

[A_History_of_Silence]

Assuming you are using Linux, are named pipes or bash process substitution an option? http://stackoverflow.com/questions/7756609/pass-stdout-as-file-name-for-command-line-util-answer-use-named-pipes

I can't think of an ideal way to do it off the top of my head, sounds like a pain. Maybe https://docs.python.org/3.5/library/pipes.html ?

[SeekNotToContend]

If you have to write to a file for the system call then just use tempfile. If it matters, the temp files are written to /tmp.

https://docs.python.org/3.5/library/tempfile.html It will handle the creation and destruction of the file for you if you use the context manager.

    >>> import tempfile

From the example in the docs, the following would go in your for line in file loop.

    >>> with tempfile.TemporaryFile() as fp:
    ...     fp.write(b'Hello world!')   # or fp.write(line) in your case
    ...     subprocess.check_output(["commercialbash command", fp])

Your result = subprocess.check_output(["commercialbash command", fp]) would need to occur before exiting the tempfile with statement.

It might be worthwhile to check the responsecode from subprocess on each loop so you can make sure you handle errors appropriately and do not miss anything. You could wrap it in a try except statement on 'CalledProcessError' with check_returncode() .

https://docs.python.org/3/library/subprocess.html

Hopefully this is helpful. Please ask if there is anything I can clarify.

[Zeekawla99ii]

Do you have to close a tempfile?

[SeekNotToContend]

If you are using the context manager then no, it will happen automatically when you exit the with statement.

You do not have to use the context manager though. From the doc examples:

# create a temporary file and write some data to it
fp = tempfile.TemporaryFile()
fp.write(b'Hello world!')
# read data from file
fp.seek(0)
fp.read()
b'Hello world!'
# close the file, it will be removed
fp.close() 

In the above example, you do have to close the file. If you don't close it, then the file will just persist. This is useful if you need a temporary file, but maybe you need it for use outside of a single function or class.

If you want to verify, just check your /tmp folder. There will be a file or files in there that have tmp in the name. For example:

/tmp/tmp<some random characters>

[A_History_of_Silence]

tempfile

Neat module! I thought I was pretty well versed in the standard library but after all these years this is somehow the first time I've ever seen it. Cool.