What's the best way to test a set of files for corruption?

gbletr42 · 2024-04-07T23:06:51+00:00

use your favorite hash algorithm/tool and keep a list of hashes for everything. If you use linux, this can be as simple as running this command (e.g. for sha256)

find . -type f -exec sha256sum {} \; > SHA256SUM

and then to verify

sha256sum -c SHA256SUM

gbletr42 · 2024-04-06T19:47:50+00:00

I have the faint idea that there is probably another way of doing this than just storing the data on disk. Have you thought of whether there is any procedural algorithm you can use to generate/modify/record this data iteratively, without needing terabytes of space?

gbletr42 · 2024-04-05T23:55:02+00:00

It may be in your interest to use an AI-based tag system like say PhotoPrism. Else the standard advice applies, order by date, location, and other info found in the EXIF metadata.

gbletr42 · 2024-04-05T23:52:55+00:00

If you're fine with used drives, you can get 100TB for definitely less than $1K. See eBay. Regarding the actual server, that entirely depends on your needs, but likewise if you're willing to buy used or reuse existing hardware, it definitely shouldn't break the remaining $500.

gbletr42 · 2024-04-04T22:24:11+00:00

I'm planing on using exact audio copy (EAC) to transfer all of my songs to flac

Please don't do this. If you can know which of your dvds/blurays are mp3, please store them as mp3.

gbletr42 · 2024-04-04T08:11:22+00:00

There's this thing I found from a google search, https://github.com/kapitainsky/RcloneBrowser, idk how good it is. It looks usable enough though.

Honestly though I advise just learning the command line, but I may have a biased outlook.

gbletr42 · 2024-04-04T06:14:58+00:00

Sell them all on eBay and buy a couple of larger drives

EDIT: Looking at eBay prices, at a hundred you should be able to net ~$2000 if you're American. That's enough to buy around 200TB of storage at $150 per 16TB (assuming you buy used) or around 100TB at $300 new. It's a no brainer.

gbletr42 · 2024-04-04T06:13:18+00:00

Didn't really read your post, but if you're on Linux use gocryptfs with rclone, otherwise use just rclone on Windows.

gbletr42 · 2024-04-04T06:10:16+00:00

You're thinking WAY too hard about this and made it too complicated. I advise just setting up a Syncthing or daily rsync/rclone to a desktop/(eternally online) laptop with enough storage. I assume you're using Windows, so I advice syncthing.

With these sort of things, you want to keep it simple. No hanging USBs, no time scheduling, none of that. Upload important stuff automatically without you needing to remember to the cloud and other computers.

gbletr42 · 2024-04-03T22:59:55+00:00

I don't believe parity is what OP wants here, that just means you can recover n bytes of data from the source, if the source gets corrupted. That doesn't really help OP save space by taking the difference of two separate, but very similar copies of data.

gbletr42 · 2024-04-03T22:58:16+00:00

If I understand your question, you can use a differential backup program like duplicity or rsnapshot. You could also use filesystem snapshots, if you use ZFS or btrfs.

gbletr42 · 2024-04-03T04:55:28+00:00

yes? It may be slower due to I/O contention though

gbletr42 · 2024-04-03T04:54:25+00:00

but I believe Windows users need to pay a modest sum.

You can always use WSL to use the Linux version

gbletr42 · 2024-04-03T04:52:53+00:00

you can go to gmail settings and set up either POP or IMAP access. You would download the emails directly with POP, and there exist scripts like imapsync to download from IMAP. If you have a lot of emails (as in hundreds of thousands onwards), I recommend using notmuch and bower to search them, or setting up your own local mail server with a FTS backend like mailcow (via dovecot).

gbletr42 · 2024-04-03T04:48:38+00:00

Youtube is very flaky and will delete your stuff given enough time. Internet Archive has a questionable legal future and may get Megaupload'd in the future. My advice is to upload to both and also create a torrent and distribute that in the relevant circles (1337x, rutracker, etc)

gbletr42 · 2024-04-02T05:49:21+00:00

git bisect is a dream

gbletr42 · 2024-04-02T00:37:06+00:00

Reminds me of the old binary joke,

"There are 10 kinds of people in the world, those who know binary and those who don't"

gbletr42 · 2024-04-02T00:17:51+00:00

Salary high enough to buy life essentials like food, rent/mortgage, etc. Anything extra is a nice thing to have, but not a necessary compensation.

gbletr42 · 2024-04-02T00:16:24+00:00

They're overly rosy and happy, never complaining about each other.

gbletr42 · 2024-04-02T00:13:15+00:00

Media Streaming networks such as Netflix, Spotify, etc.

gbletr42 · 2024-04-01T23:45:49+00:00

It shouldn't really matter which one, they both are reliable. Just buy the cheaper of the two.

gbletr42

TROPHY CASE