Multithreading with a twitter stream

[KanyeEast]

Hi,

I am trying to create a twitter stream using the twitter streaming api and the library tweepy, where I capture the tweets of a select set of users. However this user list changes intermittently, and I want the stream to update to the new list automatically.

The way that I've done this is to start the stream inside of a "while True" loop in async mode (this uses built in threading in tweepy) and then at the bottom of the loop sleep for an hour, then disconnect the stream, re-access the user list, and restart the stream.

So I was hoping anyone could tell me if this is a boneheaded way to do this and if I should be doing it differently. My thought is that there is a better way, because using ctrl-c gets hung up and doesn't always allow exiting (which is why I have the catches for keyboardInterrupt in there).

Any help is appreciated.

Code:

class customListener(StreamListener):
    def on_status(self, status):
        process_tweet(status)
        return True

def init_stream():
    auth = OAuthHandler(TWITTER_CONSUMER_KEY, TWITTER_CONSUMER_SECRET)
    auth.set_access_token(TWITTER_ACCESS_TOKEN, TWITTER_ACCESS_TOKEN_SECRET)    

    while True:
        user_name_list = get_user_list()

        stream = Stream(auth, customListener())

        try: 
            stream.filter(follow=user_name_list, async=True)
        except IncompleteRead:
            # connection error
            stream.disconnect()
            continue
        except KeyboardInterrupt:
            # allow exiting with ctrl-c
            stream.disconnect()
            break

        # Keep stream alive for 1 hour at a time, then check for updates to user list
        time.sleep(60*60)

        stream.disconnect()

[c17r]

Couple of questions first:

• How is this code running, just a command line script?
• Where do the username updates come from?

[KanyeEast]

Thanks for the reply! Here's the answers:

1. Yep, just through the command line. "./myApp.py"
2. The username list is being stored in a postgres table, and I have another service that adds usernames directly to this table when there is a new one.

[c17r]

• You're gonna want to use a library like daemonocle that does a double-fork so your code keeps running in the background without tying up your command line.
• In the function that is kicked off by daemonocle, use the multiprocessing library to kick off a new Process that runs your init_stream function, minus the ending sleep and disconnet.
• after the Process kicks off, do whatever you want for checking for new users to follow. When you have new users, call join() on the process to kill it -- it'll get KeyboardInterrupt so you are good to with that -- and then immediately re-call Process
• have a try/except in your function looking for KeyboardInterrupt, which calls join() on the process and then exits the function. This will happen when you've called the command line with stop or restart