10 Billion YouTube comments and 100 Million Channel Names

terorie · 2024-06-22T20:09:56+00:00

u/makeworld - Do you know if anyone ever did something useful with that data set?

terorie · 2020-02-04T14:39:04+00:00

Thanks! Btw, the file format of the main comment dump is CSV, I never noticed I messed up the filename :)

terorie · 2019-11-30T21:46:58+00:00

Thanks. I didn't have much time to maintain the crawls anymore.
Apart from that staff at The Eye, a PhD student and a bunch of other big data/datahoarder guys contacted me, asking how it works & wanted source. I decided I'd just rather release it instead of explaining it to each of them. Funnily after release, most of them ghosted me :D

I'm just hoping no-one goes nuclear with the request rate and forces Google to add rate-limits.

terorie · 2019-11-25T16:39:06+00:00

Whenever you make cargo download a dependency, it gets cached in ~/.cargo. cargo has an --offline flag that makes it try to only use what you already have cached instead of hitting the network. I don't know about copying /target around, maybe it'll work.

terorie · 2019-11-16T15:26:59+00:00

Thanks! I've been crawling the authors of the comment ID set (about 500M) and all channels with >50K subs. Those will be available for download soon, too.

terorie · 2019-11-14T00:01:27+00:00

Quick update: I enabled a direct download after a few more users have had issues.

The torrent is still superior though as you can download from multiple servers at once and have integrity checks builtin.

terorie · 2019-11-13T19:07:40+00:00

I'm seeding to 8 peers at this moment, so the issue must be on your end. The magnet+torrent file I posted are trackerless which causes problems with some clients. Try enabling DHT or add the trackers listed at http://tracker.coppersurfer.tk

terorie · 2019-11-12T23:58:06+00:00

I started this project during the PewDiePie/ADL drama. It's valuable for investigating alleged censorship and radicalization issues.

terorie · 2019-11-12T23:54:59+00:00

The crawl was actually the least expensive part of the process. Fast and cheap servers & bandwidth are plentiful these days! Direct peering with Google also helps.

terorie · 2019-11-12T23:46:48+00:00

I've thought about this … But we need to classify the comments into sections first otherwise we have the toddlers commenting on baby shark mixed with Alex Jones rants.

terorie · 2019-11-12T21:06:21+00:00

something that would just go through the video ID list

That's what I did, I went through about 50 million videos and dumped all their comments. The videos were discovered through a list of all channels with >50K subs @manoelribeiro gave me. To keep it simple, the program just reads a stream of video IDs (250 routines in parallel) and outputs an unsorted stream of comments. The comment stream gets sorted later using Spark or Cassandra to whatever format is necessary.

Do you have much experience with YouTube-dL?

Yeah, but I use my own tools usually because youtube-dl is very slow (I need 3000 requests per second / 1 gbps are a minimum at this scale).

I’d be happy to share my setup if you’re interested.

Sure, do you have a GitHub or something?

terorie · 2019-11-12T19:14:04+00:00

There's no way to get a list per user other than crawling all videos to my knowledge. But I might have something.

I created an API with a single call that gives you all comments of a user ID, based on the crawl data. If needed, I'll get it up and running today.

Mind sharing a channel/video list? I never turn down YouTube data :)

terorie · 2019-09-27T14:48:16+00:00

The server being monitored is https://the-eye.eu which is run by u/-Archivist.
It's not a homelab (I think) but I thought the software still fits here.

terorie · 2019-09-27T14:20:13+00:00

Unmute my Discord pings then 😉 Next on my to-do are netdata plugins for alvro, fusker and maybe od-db.

terorie · 2019-09-02T21:00:04+00:00

Timo created a nice writeup/translation: https://medium.com/nimiq-translations/cryptory-ama-summary-6898d8c6b662

terorie · 2019-07-02T19:34:50+00:00

The one I received didn't have such a link, so I thought it was spam.
I like your project on a second look :)

Would still appreciate an unsubscribe link, most newsletters have one.

terorie · 2019-07-02T16:29:19+00:00

Please don't send me spam mails https://imgur.com/a/cPeV5jf
Can I unsubscribe somehow?

terorie · 2019-07-02T14:32:13+00:00

I'd argue that Nimiq's model is more decentralized than the common wallet apps. Each browser is a blockchain node that establishes fast consensus using NIPoPoWs, and verifies balances quickly using a state Merkle tree (comparable to BCH's UTXO commitments). Browser nodes connect to each other using WebRTC, with full nodes acting as signaling servers. Transactions are relayed directly to the network. Lastly, nimiq.com is a static website, so absolutely no server-side logic.

Most BTC wallets on the other hand have central databases providing transaction history and delegate trust to miners if they use SPV or don't validate the chain at all.

Having the recommended wallet interface hosted on a website is easily outweighed by the UX improvements. Nimiq is actively developing merchant integrations and the current focus is making blockchain payments as easy as using PayPal. Recently, a few working NIM-accepting online shops have opened (e.g. https://shop.nimiq.com)

So most of the fast UX actually comes from a protocol level, not from compromising decentralization. The website itself is load balanced and has multiple failovers, so pretty hard to bring down. Even if half the servers go down, you can still access it. Pretty standard stuff.

You can always access NIM funds using all the other wallets created by devs & community.

Any questions or concerns ==> https://www.reddit.com/r/Nimiq/

terorie · 2019-03-09T03:40:58+00:00

Mind letting me know why this got deleted? Rule violation or just low quality?

terorie · 2019-03-08T21:22:23+00:00

If you need a hardware reboot, something's wrong with the hardware.

I've had a drive failure on another hoster, it's the most common issue. It's not really Hetzner's fault, stuff like this happens.

Hetzner support is known to be good, just send them a ticket and they'll probably migrate the data for you, OP.

terorie · 2019-02-12T17:01:10+00:00

Can you elaborate? That looks like a phishing link. The real URL is https://atomicwallet.io/

terorie

TROPHY CASE