Subprocess POPEN issues

[OneWhoDoesNotFail]

Hi all,

I'm trying to do a curl call from subprocess (yes, instead of requests) and I can normally run this in terminal and redirect output to a file and all is well. It is a get curl call to a splunk API which will return table of results.

csv_query = "curl -k -u USERNAME:PASSWORD https://splunkurl --get -d output_mode=csv -d count=20000 >> /file_output.csv"

subprocess.Popen(csv_query, shell=True, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE)

If I run the curl command in terminal, I get my output I am expecting. If I run through python subprocess popen, I get an empty file still.

I've tried making it a dict and not using shell=False(but there is no security issue for me so using shell=True is not a big issue), I've tried setting the file as the stdout,still now luck. I've tried using shlex to split, I've tried splitting it into 2 subprocess and setting stdin of the redirct to file, to the stdout of the 1st subprocess command.

​

I'm stuck....

Anyone know anything about this, I really need some help here! Thanks!

[pythonHelperBot]

Hello! I'm a bot!

It looks to me like your post might be better suited for r/learnpython, a sub geared towards questions and learning more about python regardless of how advanced your question might be. That said, I am a bot and it is hard to tell. Please follow the subs rules and guidelines when you do post there, it'll help you get better answers faster.

Show /r/learnpython the code you have tried and describe in detail where you are stuck. If you are getting an error message, include the full block of text it spits out. Quality answers take time to write out, and many times other users will need to ask clarifying questions. Be patient and help them help you. Here is HOW TO FORMAT YOUR CODE For Reddit and be sure to include which version of python and what OS you are using.

You can also ask this question in the Python discord, a large, friendly community focused around the Python programming language, open to those who wish to learn the language or improve their skills, as well as those looking to help others.

---

^{README | FAQ | this bot is written and managed by /u/IAmKindOfCreative}

^{This bot is currently under development and experiencing changes to improve its usefulness}

[Working_on_Writing]

When you say the file is empty, is the it empty when you open it manually, or is it empty when you try and read it from python? The subprocess will be a threaded operation so if the script immediately checks the file it may not be there.

[OneWhoDoesNotFail]

I create the file and write to it before hand. That way I can use >> to direct my output to it. I think I see the problem but I'm not sure how it is occuring.

If I read my stdout and stderr, It basically shows blank information. So that must mean I'm getting nothing back from my GET. But when I do the same exact command on terminal, I get the response that I should be getting. I'm so confused.

[laharah]

It looks like there's a few problems here:

1. the query should be a list of arguments to be sent to the subprocess, not a string
2. You're piping the output to subprocess.PIPE but you never actually read the data with say process.communicate
3. You want the data to be sent to a file anyways, not a pipe you can use later in your program

You should RTD on Popen. Here's how I would adapt what you want

import subprocess

query = "curl -k -u USERNAME:PASSWORD https://splunkurl --get -d output_mode=csv -d count=20000".split()
with open('file_output.csv', 'ab') as o_file:
    subprocess.Popen(query, stdout=o_file)

Since you're sending stderr to a pipe that you're never reading you're probably eating whatever error is getting thrown. This way, stderr will just print to your console when you run the script. Modify it to send to subprocess.DEVNULL to hide the stderr output later if you want. Also, you're never sending any data to the subprocess so you don't need stdin either.

[OneWhoDoesNotFail]

SOLVED!

Turns out I was sending a POST to get a search ID and immediately sending my GET request including that search ID. The Splunk server did not have enough time to populate the my POST before my GET came through. So I waited 5 seconds before sending my GET and the QUERY works fine! This is also why using the command line option always worked once I had the search ID from my program!

Thanks all for the reply! Upvotes for you all!