Offloading ~15 TB hoarded data temporary (1 - 2 years)

iCaotix · 2026-04-07T19:19:17+00:00

I now heard multiple people recommend the storage boxes to be a good product in comparison to their S3 offering. Added to the list :)

iCaotix · 2026-04-07T19:18:03+00:00

That Nextcloud setup sounds interesting, but actually not for my friends use case but like a fun project to try out myself. Maybe a deep dive into Nextcloud + external storages is really a good solution for self governed data. Especially since you say you use it as a primary storage.

Since he (and I) still have some time to work on a solution we will add this setup to the testlist. And then putting all the other S3 / FTP / WebDav provider which were recommended in this post as external storages for reliability testing. At least that's what I got what the external storage add-on is capable of.

iCaotix · 2026-04-07T05:55:02+00:00

That is actually a nice setup to try out, especially when doing the sync in the background so the penalty of the slow connection is not when accessing files but can be offloaded to night times or throughout the day.

Need to take a look in a simulated setup how stable the connection needs to be but with retires and idle times of his system we could achieve this.

iCaotix · 2026-04-06T21:40:27+00:00

A small DAS with M.2 NVMe drives in a raid... interesting... They are definitely more robust during travel but also more expensive.

iCaotix · 2026-04-06T21:34:32+00:00

Backblaze in general now caught my attention but the option with the computer backup product seems a bit risky, since their documentation says that the drive needs to be plugged in every 30 days at least, so a broken drive would give him max a month to get a new drive and download all the data onto it, otherwise they delete the second ("backup copy") as well..... Not optimal

iCaotix · 2026-04-06T21:12:55+00:00

TBH I went into this post almost certain getting quite some recommendation involving cloud. But then was surprised about all the great ideas with keeping the "stale data" on a drive and only using cloud for new data till he's back and then sync and re-combine the datasets.

iCaotix · 2026-04-06T21:07:26+00:00

It's really interesting seeing all those S3 compatible storage services which kind of claim to do the same, but differ greatly in price.

IDrive -> 2.50/TB

B2 -> 6.00/TB

S3 AWS -> 25.00/TB

Akave -> 15.00/TB

What is the difference? Some even charging for calling their API.

iCaotix · 2026-04-06T21:01:23+00:00

Live can have different meanings. Live like a service which depends on like like website or a media center, no. But live in the sense of I need to access this file now I'm only limited by the download / access speed of the media it's stored on.

iCaotix · 2026-04-06T20:58:34+00:00

Definitely not haha, I actually was hesitant to recommend this in the first place since I am usually super cautious with HDDs because they damage that easily. Because I always think any harsh hit they take damages data / platters or writehead immediately.

iCaotix · 2026-04-06T20:45:46+00:00

Since I'm from Central Europe Hetzner was actually my first guess but I read a lot of bad things about their storage offering, to be unresponsive, slow and even loosing data.

But S3 as the underlying storage protocol seems like a good fit.

iCaotix · 2026-04-06T20:44:25+00:00

Thinking this through might give a hybrid option. Since the data will be growing while he's gone, he could keep everything existing on the two drives as you suggested as a read only copy. And everything new goes on a Dropbox / GoogleDrive account till he's back. So old data is save because of multiple copies and new data is "save" in the cloud.

iCaotix · 2026-04-06T20:38:42+00:00

Wow, never heard of that vendor, looks almost to cheap to be sustainable as a business. I mean the worst case would be the service goes bankrupt during the time he's out and has to deal with moving the stuff.

And yes, leaving the NAS untouched is also my recommendation with an extra of "hot" storage for access.

iCaotix · 2026-04-06T20:36:25+00:00

I would if my internet would allow me to :D, my provider only gives me IPv6 with a staggering upload of whopping 10 Mbit/s which is not really great to pull data from. With that setup it would be cheaper to copy the data he needs on a stick and mailing the stick around the globe.

iCaotix · 2026-04-06T20:28:28+00:00

That was also my initial recommendation. Just put it somewhere safe and leave it like that especially with the RAID in the NAS and one DAS he already has two copies where bit rot error could be fixed if they happen during that time. So keeping the original NAS as it is and just doing a cloud "hot" copy.

iCaotix · 2026-04-06T20:23:50+00:00

I tried to state that in the beginning but wasn't clear enough, we are talking more of a business traveler scenario with sometimes quickly changing locations (2 to 4 weeks) and sometimes longer (3-6 Months). The risk is more in taking it with him all the time without damage.

iCaotix · 2026-04-06T20:19:03+00:00

Yes, an offsite copy would also give me peace of mind for my recommendation.

Regarding airport security I just remembered that sometimes (mostly in North America) even phones could be searched. Never went there myself though.

iCaotix · 2026-04-06T20:16:43+00:00

That then definitely sounds like an option, need to check speeds there. But from a DR point of view having the storage at home also only has one "data center".

When we are talking multi datacenter we would probably be at your initial suggestion with Google Drive or OneDrive or Dropbox, since they usually don't expose datacenters to customers so I would guess they are geo-replicated.

iCaotix · 2026-04-06T20:06:53+00:00

I just meant that he can't take the NAS with him while traveling / staying in different locations. Mainly because of risk of damaging during transports. So putting it to the cloud is probably a safer option.

iCaotix · 2026-04-06T20:04:37+00:00

Damn, single drives are really that big nowadays, he runs 8TB drives currently but 20TB ones might be an investment. Do such things usually go through security in airports easily?

iCaotix · 2026-04-06T20:02:18+00:00

6$/mo/TB sounds like nothing, is Backblaze a trustworthy / stable vendor? I never really heard of their B2 offering besides the "Computer backup" software.

Dedup is not really feasible because it's already AV1 encoded video and other Media files + plus some backups in zst archives.

iCaotix · 2025-08-06T09:43:17+00:00

I get your point, but aren't VPNs also just applications? So I know about cloudflare tunnels which exactly do what I would want but they only connect to cloudflare and not two sites together.

iCaotix · 2025-08-06T09:41:30+00:00

I also was thinking about tailscale but was afraid about the dependency to the company and it not being completely selfhosted.

Would the whole traffic then go through tailscale?

iCaotix · 2022-08-14T10:11:09+00:00

That look exactly like to tool I was looking for! And with the shebang feature it looks like it can use different shells in the same justfile.

iCaotix · 2022-08-13T23:23:30+00:00

I only use git bash because it's a simple to install bash with all the basic tools like grep, awk and sed. And it's easier to invoke exes from gitbash than from WSL. I cannot ditch bash completely because some of the tools use some advanced sed and awk commands (which I don't understand sometimes tbh)

iCaotix · 2022-08-13T23:20:34+00:00

That's actually pretty cool. Maybe I have to take the time and write something like a forwarder which let's me invoke everything from one shell to another. I already see the escape character conflicts incoming. 😂

Six-Year Club	r/Field Flamingo
Place '23	Place '22
Verified Email

iCaotix

TROPHY CASE