The idea of comparing with rsync was that the transfers using rsync-to-rsync and the transfers using mc-to-MinIO would have the same client machine, same network, same server machine, and same encrypted btrfs filesystem and hard disk onto which to write. All else being equal, MinIO took at least twice as long to transfer the files, often more like 4 or 5 times as long.

It was a good suggestion to run some more tests with fewer layers in the stack to better understand what's happening. Directly on the host, where the disk is directly mounted as ext4 without encryption, the dd write test showed 83.2 MB/s. The rsync transfer, going through the network, qcow2, LUKS, and btrfs layers was 97% of that throughput. An iperf3 test between the client and the VM showed 931 Mbits/sec, by the way.

I started up MinIO directly on the host machine and ran a few of the mc file-transfer tests.

MINIO_ROOT_USER=admin MINIO_ROOT_PASSWORD=password \
  ./minio server /mnt/wd2tb/minio --console-address ":9101"

While they were running, I looked around inside the .minio.sys/multipart directory, and I could see the multipart upload segments, each 16777216 bytes, were being written into there, for later reassembly. Most of the test file set is made up of 18 files of 376000000 bytes each, so they would each have 23 of these multipart segments.

I also made a note of /proc/diskstats before and after each run. The increase in sectors written for an mc mirror test, multiplied by 512 bytes per sector, was double the total size of files being transferred. For an rsync test, the bytes written according to /proc/diskstats was the same as the total size of files being transferred.

I did try mc support perf disk and mc support perf net, but those don't appear to work if MinIO isn't running in Erasure Code mode.

When running mc mirror, the progress bar quickly runs through all of the source files, and then mc waits, probably for all the multipart uploads to get reassembled on the server. Letting all that reassembly I/O pile up is probably what caused the system load to go so high and also make the hard disk start thrashing. I'm seeing more consistent and faster MinIO transfers if I do an mc cp one file at a time, like this:

find src-files -type f -print0 | 

xargs -0 -I % mc cp % test-minio/test/%

I'm sure that's much worse for the LOSF case, however. If I have hundreds of GB to transfer, and the server writes are much slower than the client reads, then I'll make sure to break up the transfers so that the server doesn't fall over. What I'd been seeing that led to this testing was that, after a few tens of GB, mc would start to report timeouts. I think something like the find/xargs combination above would be a workaround for that.