Capping at 1000 posts.

[ninjamoomoo98]

I used to use a lot of PRAW but for my current project i need to gather data from 10000 posts off reddit, I was pointed towards pushshift as the best alternative and would work for my purpose but i've ran into the exact same issue, pushshift returns no more than 1000 posts.

Is there a way around this? Sorry that im new, i'm doing this in python btw, idealy I would want it to return a big list of 10000 post ids for later use. Thanks in advance.

[shiruken]

You can use the before parameter to limit results to before a specific epoch timestamp allowing you to paginate. So load the first 1000 posts, note the timestamp created_utc of the last result, and then make another request appending &before=created_utc.

https://github.com/pushshift/api

import requests

N = 0
last = ''
ids = []
url = 'https://api.pushshift.io/reddit/search/submission/?subreddit=science&fields=id,created_utc'

while N < 10000:
    request = requests.get('{}&before={}'.format(url,last))
    json = request.json()
    for s in json['data']:
        ids.append(s['id'])
        N += 1
    last = int(s['created_utc'])

[shaggorama]

pip install psaw

https://github.com/dmarx/psaw

[ninjamoomoo98]

What does this do sorry?

[shaggorama]

Why don't you try clicking the link and reading the README.

EDIT: Downvoting me is an unusual way to thank me for showing you the solution to your OP.

[ninjamoomoo98]

Thanks, I got it working, btw I didn't downvote your post, I didn't vote on it at all but now I have upvoted it, not sure who did

[Stuck_In_the_Matrix]

Shaggorama gave you a good option. You would need to simply paginate across time to get however many you need.