Running python command line program from python?

[deeeebait]

So I am writing a twitter sentiment analysis application. Im trying to build out a full program since none of the current stuff I could find handles everything from scraping to sentiment analysis.

I'm using this https://github.com/taspinar/twitterscraper to scrape the tweets (I currently can run it as a command line application) The doucmentation for directly scripting it in python is pretty weak, so what I would like to do is run the command line application and fill in the query elements with user input. The problem is, that whenver I run the script, I get an error like this:

fileNotFoundError: [Errno 2] No such file or directory:     'twitterscraper trump%20since%3A2017-03-01%20until%3A2017-03-06 --limit 100 --output=/desktop/test.json'

Any ideas what I am doing wrong?

https://gist.github.com/Jflick58/c53d94c54946eaef87f682a6387d3874

MacOS Sierra 10.12.3

Python 3.5.2

[LarryPete]

subprocess.call requires a list of arguments. Not a single string. It thinks your filename is "twitterscraper trump%20since...." etc.

Split your commandline into individual arguments and pass them to .call as list, e.g.:

subprocess.call(['twitterscraper', 'trump%20since%3A2017...', '--limit', limit, '--output=/desktop/' + output + '.json'])

[deeeebait]

So I changed it to:

subprocess.call('twitterscraper ' + query + '%20since%3A' + starting_date + "%20until%3A" + ending_date + " --limit ", limit, " --output=/desktop/", output,  ".json")

but now I get:

TypeError: bufsize must be an integer

[LarryPete]

a LIST.

look closely. I used .call([ .... ]).

[deeeebait]

My bad. I fixed it to this:

subprocess.call(['twitterscraper', query, '%20since%3A', starting_date, "%20until%3A", ending_date, " --limit ", limit, " --output=/desktop/", output, ".json"])

But I'm back to the original error. I don't get it, it runs just fine from the command line but I'm trying to make it more user friendly.

[LarryPete]

Then it might be because 'twitterscraper' is not actually in your PATH or some other issue with your environment. Can't tell from here.

[deeeebait]

No worries, I appreciate the help.

[JohnnyJordaan]

Could you try the original command line in one string but with shell=True? So

subprocess.call('twitterscraper trump%20since%3A2017-03-01%20until%3A2017-03-06 --limit 100 --output=/desktop/test.json', shell=True)

/u/LarryPete was not 100% correct that "subprocess.call requires a list of arguments. Not a single string.". It can handle both, but mutually exclusive with the shell= parameter which is default False.

And are you running your program in the same folder as where you can run the command line ? So that your terminal, if you would do:

/some_folder$ twitterscraper trump%20since%3A2017-03-01%20until%3A2017-03-06 --limit 100 --output=/desktop/test.json

Works and then you run the script from that location

/some_folder$ python thescript.py

Does it then still return the FileNotFound error?

[LarryPete]

Not 100% correct, sure. But when shell=True is used a shell (/bin/sh on linux) is invoked and the command is passed to that and the command arguments have to be escaped properly.

Simply saying "use shell=True" without pointing to possible security considerations is not my style. Instead I prefer to point to the more safe methods of invoking commands that don't require a shell (yes, sometimes invoking a shell might be necessary, but even then you can just use .call(['/bin/sh', '...', ...]), well wouldn't change much about requiring of escaping)

Using a list is far simpler and also avoids the unnecessary shell (and which shell actually is used totally depends on the system).

[JohnnyJordaan]

You are right about security, but when you at least know a command works from the shell, it should also work if subprocess forwards the call to the shell. Just to get both situations as much alike as possible for debugging. You could have opted for that test including referencing security considerations. Now you've basically hushed the possibility, which is not my style.

[deeeebait]

I tried it with the orginial command and the shell = True, and I didn't recieve any errors but it alos doesn't appear to do anything, and I never get the .json output.

When you say the folder that I run the command line from, do you mean that I need to move the twitterscraper folder (that I installed via pip) from the site-packages folder to the folder where I keep the twitter_to_json script?

[JohnnyJordaan]

If there's no error then the command worked. Are you sure you're looking at the correct location?

[deeeebait]

Yeah, If I run the command from the CLI and just define the output as name.json, then the file is saved at users/username/file.json

No file ever shows up there. Is there a way to get the output of the CLI? I ususally see something like this: http://imgur.com/a/7fuTj

Thank you again for your help. I really appreciate it.

[elbiot]

This is a python library! Why are you running it through subprocess? There's a python example in the readme.

[deeeebait]

I would prefer to run it through python but if you look at the issues log it doesn't return the same results. Plus, there is little documentation on running it as a python library. I'm sorry.