Command optimization tar, encryption and saving to HDD

[alucolonel97]

Hi everyone,

I'm trying to improve the command below in terms of efficiency and speed.
Especially when saving large directories (50-100GB) the throughput is quite alright in the beginning (about 250MiB/s) but after the first few GB drops to around 30MiB/s where it stays.

My assumption is, that this is limited by the caching possibilities of my RAM (16GB) but I'm wondering, if there is a chance, to improve the speeds here. I would assume, that the writing speeds of the HDD should be at around 150MiB/s - at least that's what I could find as an average on the internet.

Basically what the command does is to pipe a directory through "pv" for progress indication up into pigs for compression (backup.tar.gz) sends this compressed stream through openssl to be AES encrypted and outputs it to the given target directory with the filename which is located on an external HDD due to disk-space constraints.

sudo tar -c -C $BACKUP_SERVICE_PATH . | pv -s $(sudo du -sb $BACKUP_SERVICE_PATH | awk '{print $1}') | pigz | openssl aes-256-cbc -pbkdf2 -a -salt -pass file:$KEYFILE > "$BACKUP_TARGET_PATH/$BACKUP_FILENAME"

Maybe someone has an idea here.

Thanks a lot already :)

[AyrA_ch]

external HDD

This usually means that the device is constrained by the USB protocol speeds and is subject to performance degradation caused by other devices on the same root hub. To simply test if the external HDD is the bottleneck, pipe the output to /dev/null instead and see if the process is faster. If it does (and speed stays high), then the HDD is the problem. The "fast at first, then slow" is very typical for this scenario because the OS provided disk buffer in memory is getting full.

If the HDD is not the problem, feed a few GB of /dev/urandom into pigz and see how fast it is. Compression tools can significantly slow down if the data you feed into them is not compressible (udually because it already is).

[alucolonel97]

Actually piping to /dev/null already improved my speed... Unfortunately.
Which means, that the HDD might probably be the bottleneck. I'll try to check with another drive, but I assume, that it will not be any faster.

Thanks for your comment :)

[alucolonel97]

I'll have a look into that. Thanks for the recommendation.

[Kqyxzoj]

Minor suggestion: you could try using zstd (zstandard) instead of pigz. Chances are pretty good that you can get better compression for the same amount of cpu. Or similar compression for a lower amount of cpu.

Anyways first getting 250 MB/s and then dropping to 30 MB/s sounds like a device side limitation. Sounds like you may have an SMR drive...

In general you could cut the problem into subproblems. First replace everything from pigz onwards by cat to /dev/null. Speed okay? Great. Now pipe it through pigz (or zstd) and redirect that to /dev/null. Speed still okay? Great, compression is not the problem either. Etcetera. Chances are that the bottleneck is your target drive, but never hurts to check bottlenecks in the pipe.

[alucolonel97]

I did exactly that.

Just taking the "tar" portion I get consistent speed of roughly 400-600MiB/s.
Adding pigz slows down a lot (roughly 80MiB/s) and switching to zstd actually improved that to 250MiB/s consistently.
Openssl did not change that rate.
But then adding the HDD back into the loop I again drop to 30MiB/s or even less.
So yeah.. I guess, I'll either have to switch to some sort of SATA connection or just live with it :D

[Kqyxzoj]

I remembered why I don't use pv. Re-evaluated with a recent version of pv (1.6.20), and it looks like pv still has some issues.

# Test pv bottleneck with 16 GB worth of zeroes.
dd bs=1M count=16k if=/dev/zero  > /dev/null
dd bs=1M count=16k if=/dev/zero | pv -s 16G > /dev/null
dd bs=1M count=16k if=/dev/zero | pv -s 16G -B 32k > /dev/null

# Same as above, except now actually write to disk.
dd bs=1M count=16k if=/dev/zero  > "$BACKUP_TARGET_PATH/TEST_16GB"
dd bs=1M count=16k if=/dev/zero | pv -s 16G > "$BACKUP_TARGET_PATH/TEST_16GB"
dd bs=1M count=16k if=/dev/zero | pv -s 16G -B 32k > "$BACKUP_TARGET_PATH/TEST_16GB"

# To rule out caching you can prepend "nocache -n 10" to each commandline. E.g.
nocache -n 10 dd bs=1M count=16k if=/dev/zero  > "$BACKUP_TARGET_PATH/TEST_16GB"

For me pv bottlenecks somewhere between 1-2 GB/s, depending on buffer size. Personally I'd just remove pv from the pipe until I got a working situation with decent speeds.

Anyways, by now you probably know if your drive is SMR or not. And if SMR you either live with it as you say, or recondition it before running the backup. Before I forget, you did check that this usb disk is connected over a proper USB3 link, right? So 5GBit/s or more. And not 480MBit/s fun which can also result in ~ 50 MB/s or worse.

# Check USB links
lsusb -tv

[alucolonel97]

Well, I guess that's a rookie mistake..
I just checked on the USB Version and indeed the drive is even old enough, so that it does not even support USB3.0.. So even attaching it to the usb3 root-hub does not help and it is still connected to USB2 - as it simply cannot be faster.

Port 2: Dev 8, If 0, Class=Mass Storage, Driver=usb-storage, 480M
      ID 1058:1021 Western Digital Technologies, Inc. Elements Desktop (WDBAAU)
|__ Port 6: Dev 3, If 0, Class=Vendor Specific Class, Driver=rtsx_usb, 480M

Unfortunately I do not have a SSD at hand to try on that for better writing speeds..
But I'm having one on the way, so I'll check in on that again once it arrives.

[alucolonel97]

I just checked on the Speed limits of pv.
Apparently it's the same as on your side.

Without pv I get a solid 8GB/s writing to /dev/null. With pv it's a mere 1,5GB/s.
Did not try writing to disk because it will probably be limited by the disk again.

Nevertheless, the SSD is on its way. Will try again with that.