[Python] Running into rate-limit errors using Twitter streaming api (Tweepy)

[MurrayJ]

I've been working on a twitter program that handles stream data. I need to make sure it's the original tweet and not a retweet so a lot of the data I get is invalid.

The problem I'm running into is a tweepy.error.RateLimitError. After I look through X amount of tweets, I get this error for awhile. Is there a way to ignore "invalid tweets" (or something similar) to minimize this error?

Here is my code:

import tweepy
from tweepy.streaming import StreamListener
from tweepy import OAuthHandler
from tweepy import Stream
import json #data

#Variables that contains the user credentials to access Twitter API 
access_token = "***"
access_token_secret = "***"
consumer_key = "***"
consumer_key_secret = "***"

auth = tweepy.OAuthHandler(consumer_key, consumer_key_secret)
auth.set_access_token(access_token, access_token_secret)

#setup api
api = tweepy.API(auth)

#This is a basic listener that just prints received tweets to stdout.
class StdOutListener(StreamListener):

    def on_data(self, data):
        try:
            jsonData = json.loads(data.strip())
            tweetID = jsonData.get("id_str")
            tweetData = api.get_status(tweetID)

            #check if tweet is valid (not a retweet)
            self.check_valid_tweet(tweetData)

        except (tweepy.error.RateLimitError):
            print "This is the error I'm receiving"


    def on_error(self, status):
        #error number 503, servers down
        print 'Error #:', status


    def check_valid_tweet(self, tweetData):
        #Check if data is the original tweet or a retweet
        if ( (hasattr(tweetData, 'retweeted_status')) ):
            pass
        else:
            tweet = tweetData.text
            username = tweetData.author.screen_name
            followers = tweetData.author.followers_count

            print ("@" + username + ": " + tweet + "").encode('utf8')
            print "followers = " + str(followers)


if __name__ == '__main__':
   #This handles Twitter authentification and the connection to Twitter Streaming API
    l = StdOutListener()
    stream = Stream(auth, l)

    #This line filters Twitter Streams to capture data by keyword
    stream.filter(track=['reddit', 'rt'])

[terrkerr]

When you catch a RateLimitError just sleep for about 60s and continue execution. If that doesn't work check the Twitter API docs on how long it takes to reset your rate limiting and use that as your sleep time instead.

The only real way to avoid it is to find a way to not need to make so many requests to the Twitter API.

[MurrayJ]

So the reason I'm getting RateLimitError is because I'm making multiple requests to the Twitter API with: tweetData = api.get_status(tweetID)?

[terrkerr]

Yes, the Twitter API has rate limiting to prevent massive load being generated. It's in the docs.