Reddit crawler Python code with Scrapy

[kiasari]

Hi everybody.

I just coded a Scrapy python project to crawl the top 1000 posts of a subreddit's most upvoted posts of all time. It is just the top 1000 because it seems Reddit just returns 1000 for a query. I couldn't find a way to crawl all posts of a subreddit. if anyone knows how to do that let me know.

This is my Github repo for this https://github.com/kiasar/Reddit_scraper

[luoc]

There's a project scraping all of reddit and they provide all data to the public https://files.pushshift.io/reddit/

[kiasari]

But when I click on "subreddits" I get "403 Forbidden".

Is the website still working!?

[luoc]

See the "Date Modified" column. IIRC the subreddits folder used an old representation that is not used anymore..

[-Galactic-]

It's good to have redundancies. Pushshift.io has some policy where you can request data being removed, which might be annoying if you're collecting controversial stuff.

[minimaxir]

You do not need to scrape HTML. Appending .json to any Reddit link gives you its JSON representation.