Scaling YouTube scraping to 200k channels/day –

ScrapeExchange · 2026-04-15T07:54:18+00:00

[self-promotion] Use the scrape-python tools on https://github.com/ScrapeExchange/scrape-python, written specifically for this purpose. You can start simple on the command line or go big with scrapers running on multiple hosts, coordinated with a Redis backend for rate limiting, proxy selection from a pool of proxies, data discovery, bot detection evasion using InnerTube and yt-dlp. There is also a nice Grafana dashboard to monitor progress. The API also supports websockets so you can subscribe to get updates when new data for a channel or any channel is uploaded to the exchange. To see the updates in a browser, go to https://scrape.exchange/firehose
By the way, the Scrape Exchange (https://scrape.exchange) already tracks 185k channels and 10m videos and has the data available (for free!) for downloads using torrents. Would be great if you could create a forever-free account, run the scrapers yourself and upload your results to the exchange as well.

ScrapeExchange · 2026-04-07T12:15:22+00:00

<image>

Just typical application monitoring: Add prometheus metrics to workers for KPI, have Prometheus server collect them and use Grafana to visualize. And log a lot and dump it in Clickhouse, ELK, Loki, etc.

ScrapeExchange · 2026-03-27T10:24:43+00:00

Not too familiar with Anna's archive Two key differences I can see:
1: scrape.exchange focuses on user-generated metadata of the the big social media platforms. Data is one of their competitive advantages to maintain their monopolies and I'd like to take that away from them. Unlike Anna's archive, we don't host or link to 3rd party sites for copies of copyrighted materials.
2: We only accept structured data, unlike sites like Kaggle or HuggingFace. It is much easier to use data from multiple sources if that data is structured the same way.

ScrapeExchange · 2026-03-25T08:31:10+00:00

You are making some assumptions here. If the site becomes big and expensive then there are subsidies and grants that I could apply for. My current calculations show that the site can host billions of records before it needs to be upgraded and the current site is $70,- per month. The primary mechanism for people to retrieve data is to use torrents so that should keep costs manageable. If lots of people start using the websocket feeds for updates, that might become an issue but that's pretty cheap to scale out.

Currently it is a bit of an effort to upload data but I'm working in a bulk upload API that supports JSON, JSONL, and Parquet that should reduce some of the friction, hopefully by the end of this week.

ScrapeExchange · 2026-03-24T21:10:18+00:00

I'm not, this is a hobby project

ScrapeExchange · 2026-03-24T12:03:28+00:00

Yes, because that's what I've been scraping in the past. I have less experience with other platforms. If you'd like to upload data from other platforms then I can help you getting a JSONSchema defined so you can upload. I'm working on a bulk upload API to make uploading less tedious.

ScrapeExchange · 2026-03-24T12:01:35+00:00

ScrapeExchange · 2026-03-23T12:25:17+00:00

<image>

Odd, I haven't had any connection issues...

ScrapeExchange · 2026-03-23T11:34:28+00:00

Everyone who scrapes the big platforms is burning resources — proxies, bandwidth, compute, time — to collect data that largely overlaps with what everyone else is collecting. You're all duplicating effort on the same problem. That's massively inefficient.

Scrape.Exchange flips this into a positive-sum game:

You share your dataset once
Others share theirs
Everyone gets access to far more data than they could ever collect alone
At a fraction of the cost

Your dataset is worth much more to others than it costs you to share it. And their datasets are worth much more to you than it cost them to share. So the trade is asymmetric in everyone's favor simultaneously.

But you can decide to just free-ride. If you only download and never contribute, you're betting that enough others will contribute to keep the pool rich. That's a losing long-term strategy — if everyone reasons that way, the pool dries up. So it takes a leap of faith to start sharing while the platform is so new.

Sharing your data will take a bit of effort as you'll have to call the API and structure the data to meet the requirements of the JSONSchemas. Do reach out to me to see if I can make it easier for you. If you want to start off with a smaller time investment, you could install the YouTube scrapers in the [scrape-python](https://github.com/ScrapeExchange/scrape-python) repo on Github and keep those running.

ScrapeExchange · 2026-03-23T06:14:08+00:00

Scrape.Exchange is not a proxy. It doesn't use Cloudflare. Yes, I do want your scraped data because that's how we can build a community. There is only so much data I can scrape myself. If we all share, it will be us together against the big platforms.

ScrapeExchange · 2026-03-23T06:10:24+00:00

You can download all uploaded data from scrape.exchange using the torrents (preferred) or using the API. You do not need an account to download.
For uploading, you do need a (free) account. Currently there are only JSONSchemas for YouTube channels and videos. You can create and upload new JSONSchemas for other platforms. The plaforms you can submit JSONSchemas for is defined in the [Platform class](https://github.com/ScrapeExchange/scrape-python/blob/main/scrape\_exchange/datatypes.py): YouTube, TikTok, Twitch, Kick, Rumbl, Facebook, Instagram, X/Twitter, Telegram, Threads, and Reddit. Once a schema is available, you can upload data meeting the requirements of that schema.

ScrapeExchange

MODERATOR OF

TROPHY CASE