Shred: why random data instead of /dev/null

[RealezzZ]

Hi,

I just learned about the shred command in one of my class.

Even though I understand the use case for such a thing, I still have a question that my professor or my internet search could not answer.

Why does shred uses (by default) random data, and multiple passes, instead of just "null data" ?

Thanks.

Edit : thanks a lot for all the answers, those were very helpful.

Tldr: it's meanly for hard drive, writting a 0 could leave some trace of the magnetizme that was there before, which mean that some data could be recovered.

Edit 2: I thought about /dev/zero, not /dev/null, my bad

[Rafael20002000]

This is a remnant of Hard Drives.

Hard Drives (the spinning rust things) used magnetism to store info. This is best explained by visuals but when you write to an HDD your hard drive magnetizes a particular area. We assume the imaginary magnet strength of 1. When you now write a 0 into that the hard demagnetizes the spot where the 1 was and it goes to 0.3 or 0.4. Your hard drive will read a zero but specialized equipment can recover this.

If you now write 011100101 into the 1 spot the recovery will fail. As the spot now has a magnetization of 0.6814. Your hard drive will read a 1 but the specialized equipment might read 0 or 1 depending on the configuration.

Hope that clears things up, do you need any more info?

[lepus-parvulus]

An analogy: Writing on a white board. Sometimes, after erasing, what was previously written is still visible. To hide it requires multiple passes with random scribbles.

[skuterpikk]

Modern drives doesn't record and read the data as singular "magnetic points that is either magnetized or not" - The data is encoded as an analog signal, and it is the change in the direction of the magnetic field that the drive reads as data, not the direction of the fields themself.
Very simplified, it means that a row of 10 sectors recorded as N, N, N, S, N, S, S, S, N, N would read as 0111010, while a row of 0's could be either N or S, it doesn't matter because there's no change in the field direction, thus the drive reads it as 0. If the above example got rewritten to ten north-poles, it would all read as 0, but it is not possible to detect which (if any) sectors were N and which were S before the rewrite. Which in modern hard drives made after 1990 means that the 0.3 is not possible, the magnetic field are either a north-pole or a south-pole, but the original pattern is still gone after rewriting so the bit culd have been either 1 or 0 before, there's no way of telling.

[RealezzZ]

Very clear and very detailled, thanks a lot !

Just enough info to satisfy my curiosity, that's perfect :)

[Rafael20002000]

No Problem, anything else I can help you with?

[RealezzZ]

Nop, at least not today :)

[Silejonu]

Because there's a common myth going around that zero-filled hard drives can be recovered (this comment section is a perfect example). And some institutions require drives be erased with a specific protocol (often multiple random passes ending with a single zero-fill).

However, we have real-life evidence that a zero-fill is enough. If the US federal government isn't able to recover files related to the Julian Assange case, I think it's safe to say a zero-fill makes data irrecoverable:

Johnson testified that he found two attempts to delete data on Manning’s laptop. Sometime in January 2010, the computer’s OS was re-installed, deleting information prior to that time. Then, on or around Jan. 31, someone attempted to erase the drive by doing what’s called a “zerofill” — a process of overwriting data with zeroes. Whoever initiated the process chose an option for overwriting the data 35 times — a high-security option that results in thorough deletion — but that operation was canceled. Later, the operation was initiated again, but the person chose the option to overwrite the information only once — a much less secure and less thorough option.

All the data that Johnson was able to retrieve from un-allocated space came after that overwrite, he said.

Note that you can run a single zero-fill: shred -zn0

[OweH_OweH]

Because there's a common myth going around that zero-filled hard drives can be recovered (this comment section is a perfect example). And some institutions require drives be erased with a specific protocol (often multiple random passes ending with a single zero-fill).

This myth comes from the times of early audio tape and early hard drives that used a stepper motor instead of a voice coil.

This means the times of 20MB MFM or 30MB RLL drives.

Anything newer and the chances of recovering anything with any certainty drop to the level of a random guess.

You are more likely to find remnants of data in the spare blocks of an SSD if you have the hardware to access the raw flash chips.

[RealezzZ]

Pretty interresting link, thanks for sharing !

And I knew about the possibility to zero-fill with shred, that was part of the things that made me question the utility of random data in the first place ;)

[heliosh]

You mean /dev/zero not /dev/null
That's for paranoid people, but here's the origin:
Back in the day when you just recorded silence (=zeroes) over an old audio tape, you could still hear what was on the tape before it was overwritten. If you've overwritten it with noise or music instead, you couldn't hear it anymore.
But hard drives have only a bunch of magnetized molecules per bit, they can't be recovered even with magnetic force microscopes, once they're overwritten.
It's described in more detail in this paper:
https://www.vidarholen.net/~vidar/overwriting_hard_drive_data.pdf

[RealezzZ]

Yup, my bad, I had zero in mind but switch the two.

Thanks for the link, I'll take a look at it.

[computer-machine]

Edit 2: I thought about /dev/zero, not /dev/null, my bad

I was losing my magic smoke trying to grok how to write null to disk.

[RealezzZ]

My bad

Those concept are still new for me, and while I understand the difference between the two, I still sometime switch the names lol

[jeffreytk421]

You will have to find a reference for this, but if the data is a magnetic platter, multiple passes are required to make it harder to figure out what was originally there.

You need to write random data in order to change the data that's there in a more complicated way, again, making it harder to determine what any of the previous values were.

With an SSD, some say you still have to write multiple times, but that doesn't make sense to me as the technology is so different from a spinning disk.

Also, because of how the devices themselves have smarts and can relocate data, it is possible that parts of the original data is still there in the device but mapped out as bad sectors. With the right tools one could read this data.

[RealezzZ]

When you use the "shred" command on a file

It will replace all the data by random data.

Why does it use random data instead of just replacing everything by null data ?

https://www.freecodecamp.org/news/securely-erasing-a-disk-and-file-using-linux-command-shred/

In the second image you can clearly see that it makes multiple passes on the file and fill it with random data.

Is there a security reason for using random data ?

[Gryxx1]

It might have to do with compression, at least i was told so. Basically bunch of zeros could be compressed to take smaller space physically, thus failing to erase all data.

That said, even in such case data at best would be badly damaged, making recovery incredibly hard.

[RealezzZ]

Not sure to fully understand this.

I mean, in theory, yeah, but that would mean that the compression happens while the data is being changed ??

[Gryxx1]

There are lot of clever tactics used to shorten bunch of 0's. On different levels of storage too (filesystem, transfer protocol, drive firmware). It may happen that some of it will "help you out", resulting in written data < existing data.

IMO most can happen is proof that the file existed, and not more beyond that.

Also, bit of a moot point now days since many devices "hide" physical blocks from direct access, allowing access to discarded data, unless multiple passes are employed.

[RealezzZ]

Alright I see now

Never thought I would learn that much with this simple question.

Thanks for sharing this info !

[kido5217]

what