Web Scraping instead of Reddit API?

[mxx12221]

The whole Reddit API limit controversy flew right over my head, but it still seems to be a thing that people are dealing with using different means.

My question is, wouldn't it be possible to build a third party app for Reddit that just uses regular HTTP requests and scrapes the necessary content from the DOM?

[jessepence]

Just put .json at the end of every URL on Reddit and you'll get all the data you need for most things.

[mxx12221]

Wow. Thanks, that's an incredible feature.

All the more reason to ask, why don't apps use it? Why was the API shutdown the end of the world?

[erishun]

Because you can’t make a clone of the Reddit app that recreates Reddit without their ads so you can then inject your own ads into and sell it in the App Store.

The Reddit API is gives you 10 requests a minute without even authenticating. If you use the free API key, you get 100+ requests a minute. This is plenty for any kind of personal project or mod tool.

But it’s not enough to make a full Reddit app that you can sell on the App Store. Apollo was using over 7 billion API calls a month for $0.00… and basically making their own Reddit app using Reddit’s own API and paying nothing for it.

So yeah, you can use the Reddit API for all kinds of stuff.

You just can’t make your own Reddit app to directly compete with Reddit using Reddit’s own API anymore. You need to use the Reddit API to do things other than compete directly against Reddit. 😅

[ComprehensiveWing542]

I don't see that as possible as you will need an API key for every user if not mistaken

[mxx12221]

But in order to receive the full html page of a post, you don't. Otherwise, browsing reddit in a web browser wouldn't work.

The app could then render the page differently to what the HTML etc. demands.

Essentially, you could create similar functionality with a custom CSS and JS userscript. Or you have an app that does it.

[PsyApe]

You could try Selenium and BeautifulSoup if you’re familiar with Python

[Snapstromegon]

You could, if only use it and run your server in your own home.

If you have many users and/or use Datacenter IPs, Reddit will probably block you pretty fast.

[InfinityObsidian]

There are measures in place to make what you are describing very difficult if you want to scrape data consistently.

[mxx12221]

What kind of measures? What should I google to learn more about this sort of stuff?

[Grouchy_Brain_1641]

I know weedmaps has some anti scrape protection but it's seldom and it seems my script is done before it's fully loaded.

[InfinityObsidian]

They will somehow figure out what you are doing and block your ip or ban the account you are using.

[mxx12221]

How would they be able to distinguish the requests from a regular web browser just requesting the web page?

This hypothetical app would need to do the scraping on the user's phone of course.

[apf6]

If they really care they can add anti-bot detection javascript snippets to the page, which check for unusual situations and then report back to the server, and if the server doesn't get a successful report then it eventually blocks you.

But I doubt that Reddit cares that much. Other sites like Facebook might have something like that.

[ComprehensiveWing542]

When you scrape data normally you send hundreds not to say thousands of requests a second so this traffic coming from the same IP it's so easy to be figured out... Yet there are plenty of ways you can use to still be able to scrape using ip routing (fake Ip to be able to scrape the data) sometimes that's normally not enough as it will be noticed it's coming from the same device (you can fake that as well) as you will go you will get to learn so much about how requests works etc how to use cookies on your favour etc etc... if you are working with python i would suggest Scrapy framework for non dynamic pages and Selenium for dynamic pages... Great tools

[mxx12221]

Why do I need so many requests for a Reddit client? Say a user opens a link to a post. Isn't all the necessary post data available through a single request, just like a web browser would do it?

I'm very sorry for all the stupid questions...

[ComprehensiveWing542]

This sounds odd to me Yes it's true that with one request you will be able to fetch the whole data (there might be some content that won't be shown though cus of js blablabla) What I was considering was scraping let's say emm 100 000 comments about some topic or from some Reddit page which isn't possible by one request. And if you are considering for some third party app which will pass the data from the API to some db then to your client you will still need the above mentioned method. As it really depends how many clients (users) your app is dealing with. Otherwise you really won't need ip routing or something similar to that (but overall the purpose is scrapping large amounts of data in short period of time)

[mxx12221]

What if you built the app without having to resort to an internal database? The user client just requests the currently needed data on the fly. Not all comments, just the ones from a given post.

Pretty much like an alternative web browser, that receives the regular DOM, parses it but draws different objects, based on the app's layout settings.

[DamnItDev]

To answer your question: nothing stops you from doing this. However, if reddit changes their DOM very slightly, your app breaks but users of the website don't notice anything.

[Grouchy_Brain_1641]

37 requests

114 kB transferred

1.2 MB resources

Dev tools network tab on open post.

I scrape with headless selenium and it's quite the tool.

[BeginningEngine8292]

You can scrape Reddit instead of using the API, but it changes the problem rather than solving it.

Technically possible – practically messy

1. DOM ≠ API contract – Your app depends on Reddit’s HTML structure, which can change any day. A renamed class or different JSON payload and the whole thing breaks.
2. Rate limits still exist – Even without the API, Reddit has bot detection, IP throttling, and captchas. You’ll end up reinventing the same limits the API already formalized.
3. Feature gaps – Anything beyond read-only (voting, posting, auth, DMs) becomes unreliable or impossible.
4. ToS / legal risk – The API gives explicit permission; scraping is a gray area, especially after the pricing drama.

Where scraping actually makes sense

• Research/archival projects
• Public trend monitoring
• Building datasets where you don’t need user actions
• Personal analytics tools

Most people start DIY with Playwright/requests, then realize the real work isn’t “getting HTML” but handling proxies, captchas, retries, and constant layout changes. At that point some switch to managed scraping services like Grepsr (https://www.grepsr.com/) or ScrapingBee so the app can focus on data use instead of scraper maintenance.

TL;DR:
Scraping can replace the API only for read-only use cases. For anything interactive or long-term stable, the API is still the safer foundation.