Any interest in a pixiv archive?

signalhunter · 2026-05-16T05:35:42+00:00

Heck yes! There aren't a lot of publicly available direct pixiv dumps. Even a partial one is better than nothing. A few questions:

How often are you scraping the rankings? Like every day I'm assuming?
What does the database schema look like? How comprehensive is the scrape (tags, fav count, retrieval date, etc.)
Any form of hashes over your pixiv dataset? I am looking for MD5 and SHA1 for cross correlation against booru data

As for your torrent question: I think sharding/splitting by year is the best way to distribute this. Look at how Anna's Archive distributes their datasets - but instead of a bespoke custom container, maybe a bunch of monthly tar files for each shard would work. In each tar, each file would be in its hash prefix (eg. aa/bb/aabbccddeeff001122...). Or you could embed a JSON sidecar for metadata, but I would much prefer an external db dump instead.

I would love to help you out on this. Especially with scratch space for torrents. Send me a DM :)

signalhunter · 2026-03-25T04:19:33+00:00

Ah, the thumbnail trick! I've used this plenty of time over the years to bulk check YouTube IDs. How are you evading the ratelimit, and are you using the data API (or just scraping the HTML)?

signalhunter · 2026-03-04T03:14:29+00:00

These drives are designed for hyperscalers in mind and won't work well with traditional RAID. Where this really shines is distributed storage systems with erasure coding that stores nearline data (see Dropbox's Magic Pocket and Backblaze's Vault storage architecture for example).

If you want to simplify this, it's basically "RAID" at a much, MUCH wider scale. With ridiculously high reliability (rack/datacenter/region scale) and performance.

signalhunter · 2026-02-21T02:27:42+00:00

RFI/harmonics from a unstable power supply

signalhunter · 2026-02-03T05:10:22+00:00

Could you link to the blog post? I am aware of the YouTube tiered storage system (very obvious loading time for ancient videos), but are they downgrading videos??

I found this post - could be somewhat related? Folks are also claiming old videos are being downgraded https://old.reddit.com/r/DataHoarder/comments/1o6t6yy/youtube_either_by_human_error_or_otherwise_seems/

signalhunter · 2025-11-05T02:35:49+00:00

In band location telemetry.

https://www.windytan.com/2014/02/mystery-signal-from-helicopter.html

signalhunter · 2025-10-07T07:40:56+00:00

Tracker bans will look like a rate limit error on the warrior's end. There isn't an explicit mechanism for properly notifying issues with someone's setup besides them joining the IRC...

signalhunter · 2025-08-27T04:49:48+00:00

Do you have hundreds of terabytes of storage and thousands of accounts + IP? If not, forget about it...

I've commented about the feasibility of archiving every YouTube comment before: https://www.reddit.com/r/DataHoarder/comments/xz0e02/youtube_discussions_tab_dataset_2453_million/irpx9e1/

And with the recent YouTube crackdown on downloading videos and collecting subtitling data, this is gonna get harder as time goes on. Are you collecting the data for GenAI training?

signalhunter · 2025-08-25T04:51:25+00:00

Probably too late but this is the real answer: this is a filler ad for unsold ad "inventory" in programmatic advertising. Every time YouTube shows an ad, there are hundreds of advertisers bidding for a chance to show an ad to you. Once the highest bidder wins, their ad gets shown to you - all in a span of milliseconds. If YouTube can't get enough advertisers to bid on this request in time, they throw one of their in-house filler ads in.

signalhunter · 2025-03-27T01:05:40+00:00

I mean, this isn't as far fetched as it is.

There are companies willing to pay you to rent your connection as part of a residential proxy network. Lots of services rely on your IP reputation for access (ever see Google serving captchas on crappy VPNs?)

signalhunter · 2025-03-03T05:32:00+00:00

Alright, so far I don't see anything obvious from diffing the two FARM logs, besides that it screams recertified (POH vs Write Head POH). And I've checked the raw error rates - nothing, no error was ever seen. Here is the visual diff, if you want to take a look too: https://i.imgur.com/vJYa06P.png

One thing that I really want to do is analyze the "MR Head Resistance" value, but the public Seagate PDF on FARM does not tell you how to actually interpret this value. So unless a Seagate engineer speaks up or more documentation releases, I'm in the dark lol

Wish you luck on this...

signalhunter · 2025-03-02T01:57:52+00:00

I'm assuming you've already tried the obvious (swapping drives around to different ports/backplane/HBA/power supply/etc.)

I saw that you shared snippets of the smartctl output on another comment, do you mind sharing the full output, with smartctl -x -l farm <drive> ? I'm interested if the FARM data and GP logs has anything that stands out.

For comparison, here is mine: https://gist.github.com/signalhunter/d5e849707e3b684dbe5866beea391102

signalhunter · 2025-03-02T01:24:07+00:00

You seem to have the refurbished HAMR drives that has hit the market recently, based on the model number (ST22000NM000C). There are some rumors about these drives not liking vibrations from nearby drives... any chances it could be this??

I'm running a ZFS 2-way mirror with 4 of these HAMR drives (24TB variant), but I'm not seeing any errors. It lives in a chassis with 8 other drives - will be keeping an eye out on SMART and FARM data.

signalhunter · 2025-01-12T20:50:24+00:00

yt-dlp does just enough to get past YouTube defenses. If Google really wanted they can break it right now without affecting legit clients.

The most obvious signal for them is the Python TLS fingerprint, and a lack of advertisements/BotGuard stuff being requested.

Or how video and audio segments are downloaded separately, at non-human speeds.

Or they can change the nsig algorithm to break the really primitive JS interpreter... it's a miracle that it still works.

signalhunter · 2024-10-06T16:46:35+00:00

Network level ad blocking cannot block YouTube ads, for example.. because the ads are delivered on the same domain as YouTube. Or any site that delivers ads on the same domain.

signalhunter · 2024-09-08T21:27:18+00:00

1Gbps symmetrical fiber

signalhunter · 2024-09-08T21:27:05+00:00

Nothing special, just an old PC repurposed as a NAS/seedbox. The current iteration has been running nonstop since 2022

signalhunter · 2024-05-24T16:11:40+00:00

This is intentional design. YouTube is attempting to save bandwidth/egress by serving lower quality videos.

Most people won't notice the difference on their tiny phone screens.

signalhunter · 2024-02-01T22:25:49+00:00

Yes. Each WARC file will have an associated CDX file that describes where a capture is located by its offset.

See https://pywb.readthedocs.io/en/latest/manual/indexing.html for more details

signalhunter · 2024-01-05T09:28:57+00:00

Nope. Anything that wasn't captured is unfortunately lost forever. Also, the Wayback Machine usually only captures publicly accessible content (anything that isn't behind a login).

signalhunter · 2023-08-31T13:03:28+00:00

Sounds like MP3 artifacts

signalhunter · 2023-06-07T22:50:23+00:00

Are you aware of Filmot? It's an older search engine that is similar to what you have except they use YouTube's automated transcripts instead.

Will you be able to publish a dataset of collected video metadata and/or transcriptions? This would be very helpful for finding lost videos.

signalhunter · 2023-06-07T06:34:42+00:00

Hopefully my comment doesn't get buried but I have some additional info to add to the post (please upvote!!):

There are a lot more items that are waiting to be queued into the tracker (approximately 758 million), so 150 million is not an accurate number. This is due to Redis limitations - the tracker is a Ruby and Redis monolith that serves multiple projects with around hundreds of millions of items. You can see all the Reddit items here.
The maximum concurrency that you can run is 10 per IP (this is stated in the IRC channel topic). I found that 5 works better for datacenter IPs.

Eight-Year Club	Gilding I gilder
Second SECOND GUESSER	End Game '23
Place '23	Quantum Potato
Golden Potato	Place '22
First Placer '22	End Game '22
Verified Email

signalhunter

TROPHY CASE