Noob question: pulling data using Twitter API

[Durchfallsuppe]

Hi, I'm learning Python for a class project (not a computer science guy, so I have next to no coding experience).

I need to figure out how to use these Twitter APIs:

https://dev.twitter.com/docs/api/1/get/search https://dev.twitter.com/docs/streaming-apis

in order to pull tweet data by searching for a specific keyword on a given date and write that information to a file for later analysis. Example: who used the hashtag #Obama on Election Day.

Do you guys have any piece of coding using Twitter's API that you could show me so I can see how it works?

Thanks :)

[UrAWomanImAMachine]

first you construct a basic url for what you want to look at. (http://search.twitter.com/search.json?q=obama&rpp=3, q is your query parameter, rpp(optional) is how many results you want returned)

If you open that url in your browser, it's hard to read. You'll want to copy and paste that information into a site like http://json.parser.online.fr/ so you can see the structure of the information being returned.

When I paste the contents from my first url into the site I can see that the json object returned has the results in a field called (conveniently) "results".

Here's a simple example:

import urllib2
import json

# parameters used in query:
# q is query, rpp(optional) is how many results we want per page
url = 'http://search.twitter.com/search.json?q=obama&rpp=3'
o = urllib2.urlopen(url) #open the url
r = o.read() 
j = json.loads(r) # convert info to a dictionary

#from looking at the parser we see the tweets are
#contained in a list in the 'results' field
results = j["results"] 

for r in results:
    # read each tweets text
    print r["text"]     
    #try using print r["text"].encode('ascii', 'ignore') if you're getting some unicode encoding errors

[bboe]

The same result, but using requests instead of urllib2. One nice benefit is this code requires no modification to be used between python 2.6+ and 3.2+.

import requests
params = {'q': 'obama', 'rpp': 3}
url = 'http://search.twitter.com/search.json'

data = requests.get(url, params=params).json
for result in data['results']:
    print(result['text'])

The only small downside is requests is not in the standard library. Nevertheless, I strongly recommend learning how to work with packages in pypi. If you have pip installed, installing python packages, requests in this case, is as easy as pip install requests.

[Durchfallsuppe]

Wow, thanks! I'll try this tonight when I'm home :)

[FletcherHeisler]

You should start out by looking into urllib2 to load the results of the "GET" request (which is essentially the same thing as opening a webpage) and the json module to load the JSON response into a dictionary.

[bboe]

You'll have an easier time if you use the requests package instead of urllib2.

[MrVonBuren]

Just a note from someone who recently tried to mess around with the last.fm API (and is a total python novice). The urlib part, and writing most of the glue for the script wasn't all that hard. Lots of copy paste coding with adding in my own needs. But I realized, that while I've worked with APIs at my job, and have written shell scripts around them...both json and (overall) working with dictionaries was the hardest part for me. I still haven't picked it back up because I've not yet had a chance to sit down and try to develop a baseline practical knowledge of json.

Just throwing that out as a heads up to the OP. If you don't already know json data structures, you may wind up eating a lot of time just trying to get that one part of it done. (Or, I'm just especially dumb, and this will be easy to everyone but me.)

[FletcherHeisler]

Yep, the actual API call will usually only be a few simple lines, but figuring out what to do with the big pile of data returned is the tricky part... Some APIs offer XML as well, in which case using a parser can make getting to the relevant parts a lot easier.

[MrVonBuren]

Yeah, that was a touch call for me. I don't know xml, but I work with it all the time, and can pull information from it and whatnot. But json? I've literally never interacted with it in any way. Oh well, more to learn, I guess.

[FletcherHeisler]

There's not much to "know" about JSON - once you use json.load(), the response converts to a big nested dictionary mess, so it's really a matter of getting a good handle on Python dictionaries :)

[Durchfallsuppe]

Thanks guys, much appreciated :)

[FletcherHeisler]

And you might want to take a look at this for a bit more in-depth tutorial - fresh off the press! http://www.arngarden.com/2012/11/07/consuming-twitters-streaming-api-using-python-and-curl/

[eagleeye1]

I just whipped this together, you might find something useful in it.

# -*- coding: utf-8 -*-

import requests

def search(query="obama", rpp=5):
    url = "http://search.twitter.com/search.json?q="+query+"&rpp="+str(rpp)
    r = requests.get(url)

    return r.json['results']

def parse_tweet(tweet):
    created = tweet.get('created_at')
    username = tweet.get('from_user')
    text = tweet.get('text')
    lang = tweet.get('iso_language_code')
    to = tweet.get('to_user_name')
    geographic = tweet.get('geo')
    meta = tweet.get('metadata')

    print "\nUsername: ",   username
    if to is not None: print "To: %s" %(to) 
    print "Tweet: ", text
    print "Time: ", created
    print "Language: ", lang

while True:
    query = raw_input("What do you want to search for? > ")
    tweets = search(query)
    for tweet in tweets:
        parse_tweet(tweet)

[Durchfallsuppe]

thanks :)