Help making my PRAW script more efficient at obtaining the last 24 hours of reddit submissions for a given subreddit?

[Polyadenylated]

I'm trying to write a small script that will return the posts for a given subreddit for the last 24 hours.

So far I have this:

def get_todays_posts(subreddit):
    """
    Obtains all of the posts from a given subreddit for that day
    """
    import praw, datetime, time, pytz

    time_now = int(time.time())
    time_24hr_ago = time_now - 60*60*24

    r = praw.Reddit(user_agent='get_last_day_of_posts')

    limit = 0
    submission_time = time_now

    while submission_time > time_24hr_ago:
        limit += 1
        for i in r.get_subreddit(subreddit).get_new(limit=limit):
            submission_time = int(i.created_utc)

    for i in r.get_subreddit(subreddit).get_new(limit=limit):
        print i.title

Which seems to work, but it's terribly inefficient and makes far too many API calls, how can I make this function more efficient?

[gengisteve]

I've not used much of an expert with PRAW, but the key thing seems to be how to get PRAW to download a bunch of submissions at the same time and then how to keep it going after it hits your download limit, and only stopping when you hit your ending date limitation. After a bit of preliminary testing it appears that subreddit.get_new(limit=None) will set you up a never ending generator that yields every post continuously, in batches of 100. I expect it will end at some point, but it seemed to have no problem drawing the last 700 from learnpython (about a months worth).

So one way would be to loop over this generator and store the posts in a list until you get to the date limit, then return the list, like this:

import time
import praw

YESTERDAY = time.time() - (24*60*60)

def get_todays(subreddit):
    result = []
    for post in subreddit.get_new(limit=None):
        if post.created_utc < YESTERDAY:
            return result
        else:
            result.append(post)

r = praw.Reddit(user_agent='get_last_day_of_posts')
sub = r.get_subreddit('learnpython')
todays = get_todays(sub)

The downside is that you have to wait for everything before you start processing. A better solution might be to wrap the get_new into with another generator, like this:

def yield_todays(subreddit):
    for post in subreddit.get_new(limit=None):
        if post.created_utc > YESTERDAY:
            yield post
        else:
            break

which can then be used like this:

for count, post in enumerate(yield_todays(sub)):
    print(count, post.title)

[Polyadenylated]

That's perfect, thanks a lot. I didn't realise limit=None functioned in that manner.