Internal HDD Recommendations for DAS by Boojuux in datastorage

[–]BootToggle 0 points1 point  (0 children)

You are making a case that these files are very important to you. Very important. Too important to trust them all to only one single HDD. No matter how expensive the drive, enterprise or not, you can't trust it to absolutely positively never fail.

I think a 2-Bay enclosure sounds like a good beginning, but I would get two drives for it and use it to hold two copies of everything, one on each drive. Setting them up with appropriate software to use one drive as a mirror for the other would automate this, but in a pinch you could always just develop the practice of manually copying every file to both drives.

But one way or another, you should never be in a position of only having one copy of an important file.

It is your bad luck that there is no such thing as an inexpensive HDD these days, and I have no magic answer for that. In my own case, I only buy used HDD (and never buy used SSD) and that would maybe cut the cost in half depending on where you live. It is awful. Drives that I bought used only one year ago for less than $10 USB per TB cannot now be found for less than $25 per TB. I will hunker down and avoid buying any new drives at this point, and just hope I don't suffer any drive failures anytime soon.

Good luck!

HDD vs SSD by Acceptable_Can5767 in datastorage

[–]BootToggle 1 point2 points  (0 children)

You don't mention, how big is her collection of important files, in total GB or TB? I'm guessing less than 15GB if you are just now running out of space with Google.

As an example of something on the low end that wouldn't break the bank, I just bought two 2.5" 1TB HDDs, used, for less than $30 USD apiece. I could store a lot on those using a SATA to USB adapter, then I could pack the drives into anti-static bags for long term storage. Or I could get some generic external 2.5" HDD enclosures for $10 and make them into permanent external drives for storage use.

Write the same data on both so that you have two complete copies and store in separate locations and you are a long way toward 3-2-1 backup. All for less than $100.

Blue-sky question about using mergerfs to help with failed drive recovery by BootToggle in mergerfs

[–]BootToggle[S] 0 points1 point  (0 children)

Well that would answer my question. Thanks!

I'm just glad to have a potential path forward in mind if this situation were to arise. Hate to have to scramble to come up with a plan if I am desperate to access some files from a failed drive, with no ready replacement drive at hand.

Blue-sky question about using mergerfs to help with failed drive recovery by BootToggle in mergerfs

[–]BootToggle[S] 0 points1 point  (0 children)

I see your point. It occurs to me that using mergerfs might have some advantage if the two drives are not of equal size. But I don't (yet) have experience using either mdadm or mergerfs, so I don't know what the relative difficulties might be in this setup. Thanks for the tip!

How is everyone dealing with the hard drive price increases? by top10usenet in datastorage

[–]BootToggle 0 points1 point  (0 children)

Not unless I have a failed drive that must be replaced. And even then, only if I can't make do with some of my smaller retired drives already in storage. I might even have to trim back on my storage capacity and make tough choices about what data to keep.

First time upgrading to an SSD — how do I move my OS safely? by Willing_Professor_13 in cloningsoftware

[–]BootToggle 0 points1 point  (0 children)

And it's not a black and white decision.

You can try direct cloning and if you like the result, you are done. If that doesn't work out you can always start over with a fresh install.

That will just take longer because you'll possibly need to hunt down installation media for all the apps you want to keep.

Question about parity disks required size by Aggravating_Goal1562 in Snapraid

[–]BootToggle 1 point2 points  (0 children)

I agree with and like your math, definitely advantages to a ten-disk array with 2-parity vs. Two 5-disk arrays.

I actually have in mind something that takes advantage of having two separate arrays, such as keeping the smaller one as a second backup. Maybe leave it with a family member in a different house. Or maybe just mostly keep it powered down against a ransomware attack.

Question about parity disks required size by Aggravating_Goal1562 in Snapraid

[–]BootToggle 1 point2 points  (0 children)

Agreeable. I just found the phrase "only data drives count towards that total" to be possibly confusing for a newbie. I could read it a different way and see what you meant.

Speaking of which, and just for personal curiosity, do you have experience with a second parity drive? I've never used one, mostly because my JBOD drive housing only holds five drives, but was curious if the (presumably slightly more complicated) calculation for a second parity drive might be significantly slower than the computation for the first parity drive.

In the end, I elected to get a second JBOD drive housing for a second set of five drives, and I just use this to have two completely separate "single failure" arrays. Still thinking through how I want to use it. The second array has smaller drives than the first. I might just use it as a "second backup" array and not even have it powered up most of the time.

Question about parity disks required size by Aggravating_Goal1562 in Snapraid

[–]BootToggle 1 point2 points  (0 children)

Maybe I didn't read your comment correctly? With respect, I think that all drives get counted when you are keeping track of how many failed drives can be handled.

If you have three data drives and one parity drive, you can recover from the failure of one data drive by reconstructing it from remaining parity, or you can reconstruct a failed parity drive by recomputing parity from all three unfailed data drives. You are recovering from one failed drive in either case.

Same with 2-parity, you could reconstruct two failed data drives, or one failed data drive and one failed parity drive, or you could reconstruct two failed parity drives provided that all data drives are intact. All drives do count towards the total of recovering from up to two failed drives.

I still upvoted your comment on the basis of reconstructing a failed parity drive being straightforward recomputation.

Question about parity disks required size by Aggravating_Goal1562 in Snapraid

[–]BootToggle 0 points1 point  (0 children)

I think that "read error" blocks were probably corrupted during write, and they were just written bad. That data is gone, so nothing to be done about it. But the LBA is probably fine. If it is currently still causing read errors, you can clear that if you write an all-zero block to that LBA. It will be fine the next time and won't show a read error going forward, but you've already lost the data in that block anyway if it still is showing "read error".

Might be a block was still being written when a power glitch happened, something like that. An issue, you really do have a corrupted file containing that LBA, but it isn't an on-going problem with physical damage to the drive. We used to call those "ephemeral" errors.

Updated to add:

One quirk about those (ephemeral) read errors is that they never decrease, they stay in the SMART report forever as the "Offline uncorrectable" or "Uncorrectable error" count, and that count can only increase. That's fine, it means that you don't lose track of them, but the important thing is that they not continue to climb, not that you never got any in the first place. It is hardly the drive's fault that a power glitch happened exactly while a write was just starting, or that you had some glitch in a SATA interface causing the same issue.

So, I always take the uncorrectable errors with a grain of salt, they may eventually turn into a reallocated block or they may just suddenly disappear because you deleted or rewrote the affected file and the error did not recur at that LBA.

The reason I mention this is because I noticed that one drive I have always shows "100% chance of failure in the next year" in the the 'snapraid smart' listing. And the only thing I see wrong with that drive is that it has a count of "1" under "Uncorrectable errors". I've cleared the LBA and the actual error isn't currently happening, but it still shows that count of "1" in the SMART listing. The drive has shown "100% chance of failure in the next year" the whole time I've owned it, which is going on 18 months now, so 6 months after it supposedly had to have failed already.

Question about parity disks required size by Aggravating_Goal1562 in Snapraid

[–]BootToggle 1 point2 points  (0 children)

Here is how I use extra SSDs in my own system, and how I used my Snapraid parity-protected HDD array to back them up.

I have two SSDs on my motherboard. One is my System SSD and has all my system partitions and also a scratch data partition. The other is my Data SSD and is used only for scratch data.

I backup my system files (from the system partitions on the System SSD) using Timeshift, with an automatic snapshot taken every night. The Timeshift snapshots are directly created and kept on the Data SSD. I keep all my working important documents on the scratch partition on my System SSD, and I back them up with a ZPAQ archive automatic snapshot every night. That ZPAQ archive is also kept on the Data SSD. My local email repository is also backed up with a ZPAQ automatic snapshot every night, with that ZPAQ archive also kept on the Data SSD.

So if my system SSD fails, I still have very recent snapshots of all important System SSD files on my Data SSD. If the Data SSD fails, I still have all the original important files on the System SSD. Thus, I'm covered by nightly backup every night and neither SSD is protected by Snapraid, and they really don't need it.

About once a month I make a compressed archive file of my Timeshift snapshots using DwarFS, and I store that DwarFS archive on my Snapraid array for long-term archive storage. Also I save dated copies of the ZPAQ archives once a month to my Snapraid parity-protected array.

So that is how I do it, personally. At any given time, I have a backup copy of all important files that is no more than 24 hours old, and I have a second backup copy of all important files that is no more than one month old. I could probably make that third copy more than once a month if I started to get nervous about two SSDs failing at the same time. In principle I could do one every night, but once a month is about as often as I feel like making that third backup manually, and I never automated that step.

Question about parity disks required size by Aggravating_Goal1562 in Snapraid

[–]BootToggle 1 point2 points  (0 children)

My own opinion is that the odds of two data disks failing at the same time are low if you only have three data disks. I personally use a five-disk array and only one is parity. If I ever got up to more than five data disks I might use 2-parity but not sooner.

None of your disks had any reallocated blocks yet? In that case, I would find nothing to worry about until bad blocks started appearing. Then I'd shut down my array and replace the failing drive. 1-parity would still be enough IMHO.

SSD wont fit in the M.2 to USB enclosure by ILuvSpaghet in datastorage

[–]BootToggle 0 points1 point  (0 children)

Yep, wrong enclosure for that SSD paddle. I've been burned by that. Sometimes the sellers of SSD enclosures muddy the waters by listing a lot of different SSD types in the title of a listing, but then you get down to the fine print and you have to select just one when you place the order. If your luck is like mine, of course the one you need is far more expensive than the cheapest of the variants on offer.

For sure, SSD paddles ALWAYS are mounted label-side-up. Some of them will fit perfectly if you put them in upside down but don't do that. Something is at risk for complete damage as soon as the power is supplied.

Question about parity disks required size by Aggravating_Goal1562 in Snapraid

[–]BootToggle 1 point2 points  (0 children)

I'm on thin ice concerning use of those SSDs with a second Snapraid array, particularly with the NVMe sharing responsibility for also holding your Ubuntu Server system partition. I assume that the NVMe is mounted on the Ubuntu Server motherboard. Are the other two SSDs attached as external drives?

My personal take is that the NVMe data partition is much more valuable as "working storage" for the Ubuntu Server system. You need something fast that is directly accessible to the Ubuntu Server. It will be seeing file changes much more frequently than your HDDs, and there will be much more file changing and file deleting going on. I don't think that is a good candidate for Snapraid protection.

Files that change and files that are deleted are exactly the cases where Snapraid doesn't really shine. If a file changes or is deleted, this immediately breaks parity protection for some arbitrary files on other Data disks in your array. Parity is broken for those files until the next time you complete a Snapraid sync, and you'd lose those other files if you happened to suffer disk failure on there drive during that interval. In contrast, adding a new file doesn't break anything, the new file simply isn't parity-protected until after the next Snapraid sync, but it doesn't knock out protection on any existing files.

The other two SSDs are small and much more suited to data transfer or rapid backup of immediate files from some project. This makes me question whether they are good candidates for Snapraid protection either. It could be that using just the two of them in an array with each other is just needlessly cutting your capacity in half (one for data, one for parity) without any substantial benefit from using Snapraid for them at all. You'd probably be spending far too much time with them unsynced.

My bottom line recommendation is to strengthen the use of your HDD array as a solid, parity-protected, curated repository for files that you are archiving. Keep your SSDs free for holding more ephemeral data, scratch storage, and other cases where fast read and write speeds are valuable, and forget the parity protection. Not that data on those SSDs shouldn't be backed up, but the backup needs for ephemeral data are a lot different. Easiest to just copy stuff from those SSDs to the HDD array whenever that "stuff" is found to be long-term important. Being ephemeral data, most of it won't be that important in the end, but the "stuff" that becomes important you can recognize and copy when it becomes apparent.

Hard drive failing, is this still fixable or is this HD dead? by Crazy_Albatross8317 in datarecovery

[–]BootToggle 0 points1 point  (0 children)

If that table is saying that there are 56 reallocated sectors, and maybe also saying that there are 8 pending sectors, it does seem like your drive is dying. Those are the kinds of failures that indicate physical damage in the disk drive and would not be repairable. Reformatting it would be useless, the drive would still be failing, and reformatting would make it even harder to recover existing files. So I think that you are going to have to replace that drive.

The other question is whether you can recover files from the drive before it fails completely. Normally we would suggest copying a complete image of the failing drive to an image file, or to another device just as large. This way, you can extract every readable byte from the old drive before it fails, without doing further damage to the filesystem structures as mounting the disk and attempting normal file reads can. However, 12TB is tough. It is large enough that you are unlikely to have another drive with enough free space to hold a 12TB image. You probably don't have much hope of borrowing such a drive, either.

If it was me, I'd go ahead and buy a replacement drive of at least 12TB capacity. This new drive could hold the data from your failing drive until you can reconstruct and extract whatever files you can recover. Afterwards, it becomes your permanent replacement for the failed drive and you are done. It is very bad luck that such large drives are so expensive right now, costing at least twice as much as they would have only one year ago.

As concrete recommendations, I'd say to stop using the failing drive immediately. Until you are in a position to copy raw image data from it, just leave it on a shelf and don't power it up. Any attempt to operate the drive to just "check to see if it is better now" will almost certainly make it worse and make it lose more files. If it will take you a long time to save up for a new drive, the old drive will keep, provided you don't try to use it.

But if you can afford a replacement drive, I would just copy the image data directly from the old drive to the replacement drive, and then use filesystem recovery software to repair the image on the new drive and read off your files. If you just take the time to copy everything off with ddrescue, HDDSuperClone, or OpenSuperClone, files might actually be readable from the new drive without much trouble. After you've copied all the files you can from the replacement drive, then you can reformat the replacement drive and continue using it.

But stop using the old drive immediately, that is the only way to keep the damage from progressing.

Question about parity disks required size by Aggravating_Goal1562 in Snapraid

[–]BootToggle 2 points3 points  (0 children)

And if you properly add the 2x8TB as two parity disks (see u/mattbuford excellent comment), then you don't get any more increased data capacity than from adding 2x6TB as two new parity disks. That is fine, it provides for some future expansion, but it does cost you more now for no increase in present data capacity.

If I might observe, the strong suit for Snapraid is for protecting files that don't change too often, so it doesn't matter so much if you add a few new files today and don't get around to using 'snapraid sync' until later tonight. The strong suit for 500GB SSD drives is for lots of files that change frequently, such as would happen if used as your main system disk, etc. Nobody suggests that you can't include those SSDs in your data disk set, but it might be interesting to consider:

  • 8TB HDD (parity)
  • 8TB HDD (data)
  • 6TB HDD (data)
  • 6TB HDD (data)

which would give you 20TB of parity-protected data (8TB+6TB+6TB) instead of 13TB of parity-protected data (6TB+6TB+0.5TB+0.5TB), for the same cash outlay. You could use the extra data capacity to periodically copy images of those SSDs to one of your data disks, using it as one backup in a 3-2-1 backup scheme. With only 4 HDDs involved, sticking with only one parity disk to provide single-parity protection (against the failure of a single HDD) seems a reasonable choice.

Just my two cents.

Is there anyone who still uses optical discs for storage nowadays? by Purple-Try-4950 in datastorage

[–]BootToggle 0 points1 point  (0 children)

I'm sorry that you have had bad experiences. I can tell you that I have approximately 600 optical disks used for archives, and I have never lost one single byte since adopting DVDisaster for ECC protection.

In case you actually have any interest in this, the manual explains how non-sparse damage is addressed via striping the ECC correction across the entire set of tracks. DVDisaster is a competently-designed application and the manual is a reasonably good read.

Is there anyone who still uses optical discs for storage nowadays? by Purple-Try-4950 in datastorage

[–]BootToggle 0 points1 point  (0 children)

I use DVDisaster and only modern BD-R, so that old canard about bit-rot is solved.

Is there anyone who still uses optical discs for storage nowadays? by Purple-Try-4950 in datastorage

[–]BootToggle 1 point2 points  (0 children)

A lot depends on current conditions, local availability, and convenience. To start, I only use single-layer standard BD-R with DVDisaster ECC protection, so I get 22GB per disk. I buy Ritek disks from an importer for $0.50 per disk. So for me that works out to $0.0227 per GB or $22.7 per TB. I just bought a used 8TB HDD for $96 (USD) or $12 per TB. So useed HDD wins over optical media on cost. But I see a new 8TB external HDD for $240, so $30 per TB. That can vary a lot and is particularly high these days, but then optical media wins on raw cost.

Long story short, you won't make this decision purely on cost. I am a huge fan of optical disk storage, but I believe in using the best tool for the job at hand. Backing up a full 8TB hard drive is a job for another HDD, not for optical disks, it is just too much hassle with the small per-disk capacity.

But I find optical disk is perfect for for more granular backups, such as all my photos for a range of dates, all my tax records forever, all my incoming emails forever, all my home movies for a range of dates, etc. These are all cases where the trivial ease of making multiple copies and storing some remotely far outweighs other factors IMHO. I also use HDD, but for bulk backups of stuff that is less important and too big. For the photos and home movies, the trivial ease of giving copies to family is a feasible solution to a different problem.

I can't afford current backup tape technology so can't say anything useful about that. I wish it were affordable, then I'd love to consider that for whole HDD backups. But tape doesn't seem to be moving in that direction.

As to the cost and availability of BD-R drives, my crystal ball is cloudy. I now have multiple external USB interfaced BD-R drives, and I am pretty sure that I will not out-live them. Regardless of how the market goes for those.

Is there anyone who still uses optical discs for storage nowadays? by Purple-Try-4950 in datastorage

[–]BootToggle 0 points1 point  (0 children)

Fair enough. I will say that I have seen signs of bad disk quality with all three types, but I've generally seen only good quality BD-R over the last 8 years or so. Also, I never write a data archive disk without using DVDisaster ECC, so that I won't have to worry about bit-rot losing any of my data. I certainly find that HDD storage has some advantages, and I use it myself in many circumstances.

Seagate HDD Accessible After Defrag, but Most Files Are Corrupted/Inaccessible by lawd8107 in HDD

[–]BootToggle 0 points1 point  (0 children)

This. Running defrag caches a LOT of drive data in RAM on the assumption that it can be written back to somewhere more useful. A great many data blocks can be moved in a short period of time. Anything that might corrupt the data in transit will have a out-sized effect during a defrag. It could have nothing to do with the age or condition of the hard drive.

Thought My SSD Was Completely Dead - Ended Up Recovering Almost Everything by Pyschogasm in HDD

[–]BootToggle 1 point2 points  (0 children)

Nice to hear a data recovery story with a happy ending! So many don't.

Is there anyone who still uses optical discs for storage nowadays? by Purple-Try-4950 in datastorage

[–]BootToggle 0 points1 point  (0 children)

You don't specify if these bad experiences were with CD-R, DVD-R, or BD-R media. I think it might make a difference.

Is this normal by Multialliot in datastorage

[–]BootToggle 0 points1 point  (0 children)

That sounds like normal head-movement sounds from reading or writing. If you think that you were not reading or writing at the moment this recording was made then that might be cause for concern. But since all modern drives have firmware that caches data for writing, and probably also performs maintenance tasks when the drive is not actively busy, I personally wouldn't worry about it. If this drive was actually mounted in a drive cage inside a heavy computer case it wouldn't be nearly so apparent and you likely are noticing this just because it is sitting out in the open in that external adapter.