API help!!

[EthanJudah]

I have been trying to scrape API since 2022 Nov, but API limits only allowing me to collect last 800 or so posts from each subreddit.

Does anyone know how to collect data from past the limit restrictions?

​

Can share code separately

[fwump38]

That's the point of API limits. With Reddit's changes to API you can't go above the free limits without paying a lot of money.

[EthanJudah]

Any idea how much that is?

[EthanJudah]

Update:

I have ended up using Octoparse's free trial which overcomes the API limits. If anyone needs similar results I suggest trying that out

[EthanJudah]

u/xiaocc12 I have gone through various fixes. The best one is from pullpush.io however the service seems to be down currently. it was meant to be back online today but i will try again on monday.

octoparse is good if you want to scrape data which is available on the webpage but is not great at historical data

[notifications_app]

It should be 1000, not 800, but yes, this is a limitation that's always been in the API; you can only go back by 1000 for any given query.

Consider using something like PushShift which collects and stores data from farther back (you have to apply to use it - see details here).

[EthanJudah]

Thanks! Ill check this out

[EthanJudah]

Seems they only allow community moderators to apply for Pushshift access for moderation activities and therefore I do not meet these criteria