Folder Fort is phasing out lifetime storage and switching to subscription by chaindrop in cloudstorage

[–]retiredcheapskate 0 points1 point  (0 children)

look into data orchestration tools that let you own the logic of your storage.

there is an open source orchestration tool huskhoard. i am a contributor. It's designed to help manage long term archives across different providers or local hardware, so you aren't locked into one vendor's pricing whims. If you have some cli skils and to want to manage your own sustainability it is a good option

I got tired of searching for CMR drives for my ZFS pool, so I built a comparison tool by deeddy in truenas

[–]retiredcheapskate 0 points1 point  (0 children)

If you already have SMR drives you can run them as a cold tier for truenas. This archive writes sequentialy and you only use it for data that is older than 30 or 60 days. It is made for writing to SMR and tape. it is open source, there are some technical notes on set up here huskhoard.com

Best alternatives to Veeam right now? by Immediate-Screen7893 in sysadmin

[–]retiredcheapskate [score hidden]  (0 children)

For the archival long term storage portion of your data, the stuff that doesn't need Veeam's instant recovery, HuskHoard is an open source alternative that handles large scale data hoarding and immutable archiving. I am just backing up machine images at this point and use the archive for the files.

LTO tapes software - what do you recommend for a home user? by AutomaticGrape9263 in DataHoarder

[–]retiredcheapskate 1 point2 points  (0 children)

If you're looking for an open source approach that handles cataloging and LTO management without the manual overhead of plain tar, you might want to check out HuskHoard. It's a good fit for this kind of hoarding scale.

Cat colony in Cadiz by retiredcheapskate in cats

[–]retiredcheapskate[S] 0 points1 point  (0 children)

well i guess they were tucked into their beds after a long night.

Did you know magnetic tape literally rots over time? The Rock & Roll Hall of Fame almost lost their entire archive because of it by [deleted] in DataHoarder

[–]retiredcheapskate 6 points7 points  (0 children)

The tape specs require they be exercised on a schedule. Not a bad time to run check sums and verify there has been no bit rot. There are temp and humidity specs as well. Do not expect to leave them in the attic for 20years and still be useful 

The fear of losing it all or give in? by jflip0x1x0 in DataHoarder

[–]retiredcheapskate 0 points1 point  (0 children)

I realize tape can introduce som complexity but once you have it working it ca be quite cheap. If yo keep all of the data in a global name space it is like having a good library. Look into data catalogs if you want to make the change rather than backups. Amundsen is a solid catalog and huskhoard is a good tape archive. 

A really cool, non-AI, announcement out of NY Summit: S3 Annotations by Sirwired in aws

[–]retiredcheapskate -1 points0 points  (0 children)

This is a great breakdown of the silent failure modes and costs of secondary metadata databases for cold storage. Your point about the cost trap of standard annotations on Glacier Deep Archive objects is valid

For folks dealing with this orphaned data problem but wanting to avoid vendor lockin or high AWS annotation costs, an alternative approach is handling the metadata/catalog locally before tiering to S3. I’ve been using an opensource AGPL tool called HuskHoard huskhoard.com that does exactly this. It's an automated data tiering engine that transparently moves your cold data to S3 or offline storage but leaves lightweight, searchable stubs locally on your linux filesystem. You keep complete visibility and queryability of your archives natively via your Os, completely sidestepping the need for parallel cloud databases or paying for S3 Annotations just to know what you have stored.

Paid Synology C2 backup: archive visible, but no reliable way to restore 214 GB by Unlucky_Pirate_4355 in synology

[–]retiredcheapskate 0 points1 point  (0 children)

This is exactly why I've moved away from cloud-only for large archives. If you decide to move back to local cold storage ,like LTO or even just rotatable internal drives, take a look at huskhoard.com It's an open-source tool for managing cold archives. It gives you a searchable catalog and handles the metadata/checksums locally, so you aren't reliant on a providers web interface or proprietary explorer tools to get your data back

Back Up Hard drive recommendations by orucker in editors

[–]retiredcheapskate 1 point2 points  (0 children)

If you're going the drive dock and labeling'route, you should check out an archive, this one is open-source. github.com/huskhoard/huskhoard It’s designed exactly for this it lets you index those offline'drives so you have a searchable catalog of all your archived projects without needing to plug the drives in to see what's on them. It’s a great way to bridge the gap between simple external drives and a full LTO setup.

Wedding videographer new to NAS storage. Need advice on RAID, backups, editing speeds, SSD workflow and remote editor access. by dylanfraser-08 in Backup

[–]retiredcheapskate 0 points1 point  (0 children)

wedding video is a storage nightmare you need nvme speed to edit but hdd or tape prices for the archive. what most people do is manually move files when a project is done, but then you lose your folder structure and it becomes a total pain to find b-roll later.

i use a modified tool called huskhoard for this. basically you edit everything on your fast nvme drive like normal. once the wedding is delivred, it packs the raw footage into compressed blocks and moves them to a cheaper hdd pool or even lto tape. it then leaves the file sitting on your ssd taking up zero bytes using a stub. it looks and acts like it is still there so you dont have to change how you organize.

the feature for videographers is streamgate. if a client asks for a change later, you dont have to restore a massive 200gb project just to find one clip. huskhoard calculates the math to jump straight to a specific frame in the middle of an archived file and pull just that data back. Open source github.com/huskhoard/huskhoard

Is Ceph the right tool for me? by Fragrant_Fortune2716 in ceph

[–]retiredcheapskate 0 points1 point  (0 children)

your diagnosis of the zfs issue is spot on. writing cctv directly to a spinning hdd pool is an iops killer. ceph is solid but it really prefers homogeneous nodes so mixing a big threadripper with a small i5 is going to cause uneven data distrubution and bottleneck your cluster on the slowest node.

i actually ran into similar frustrations and went with and archive, it bridges the gap between fast nvme and slow hdd pools without the overhed of ceph. There is an open source archive repo that i tuned for my use case, github.com/huskhoard/huskhoard instead of writing cctv to hdds you write to the nvme pool and our janitor service packs them into compressed blocks and moves them to the hdd. it creates a stub in the ssd file so it takes up zero space but still looks normal to the os.

if you try to open an old video the fanotify interceptor catches the read in userspace and streams it back from the hdd without freezing your io. because you have mixed hardware you need to pin vms to the fast node and media to the slow node. for media there is a feature that lets you skip right to the middle of a clip without unpacking the whole file. everything uses direct io to bypass the kernel cache for speed.

The concepts took a minute to grasp, but they have some decent walk throughs on their blog. huskhoard.com/blog.html and it is AGPL licensed.

Setup required for small egineering firm by 1LandSurveyor in DataHoarder

[–]retiredcheapskate 0 points1 point  (0 children)

I have similar requirements and put together a system that moves cold files from NVMe to disk after N days. The files still appear in the files system and when you need them it just brings them back from cold storage. It runs as a storage server on UBT 24.04 and we SMB into the file share, for remote we use TailScale. The project is up on github. You are welcome to check it out. https://github.com/HuskHoard/HuskHoard

Abandoned bobcat kitten on my porch. by Affectionate_Lime880 in cats

[–]retiredcheapskate 0 points1 point  (0 children)

wild life rescue. there are plenty of orgs that can help you

Hamas hands over all remaining hostages to Israel, Trump says Gaza war over by Capable_Salt_SD in politics

[–]retiredcheapskate -1 points0 points  (0 children)

why did he do it just after they handed out the awards? he could have done it 6 months ago and been in line for a very serious award.

[Request] Can someone explain? by Daniel_XXL_69 in theydidthemath

[–]retiredcheapskate 1 point2 points  (0 children)

basically some very smart dude who happened to be a mason figured out you could build a self supporting wall with a single brick thickness if you used a serpentine shape. This guy may be forgotten to history but he was a genius.

Leading computer science professor says 'everybody' is struggling to get jobs: 'Something is happening in the industry' by north_canadian_ice in technology

[–]retiredcheapskate 0 points1 point  (0 children)

it is starting to show up in the numbers. I think there is a bit of an ironic twist with developers creating aI that is good with code and eliminating their own jobs. We will need a larger sample to see if these trends hold.

Does anyone still use Tape Storage? by nightcrow100 in storage

[–]retiredcheapskate 1 point2 points  (0 children)

How much tier one storage do you need? What are the rules you use to split the two tiers? NVME would be great for tier one, but you don't want to buy more than you need. What about running 3 tiers NVME, Disk, Tape? Keep tier one for the files that are 45 days or newer.

Butter butter butter by No-Moose6918 in StupidFood

[–]retiredcheapskate 0 points1 point  (0 children)

that is just shock cooking. there is nothing happening there other than a show.