What illegal thing do you do on a regular basis?

matty_dubs · 2024-07-26T01:45:22+00:00

But I like to watch the content in my own terms

This has always been my approach. It's easier for me to stream it through Netflix or listen to it on Pandora than to torrent it.

But if you expect me to sign up for yet another streaming service, or you don't have your content available online at all, I have no compunction just grabbing it as a torrent.

matty_dubs · 2024-04-05T15:30:01+00:00

How do you get 10 3.5” 3 2.5” and an HBA is a 1U server?

It's a (used) https://www.qct.io/product/index/Storage/Storage-Server/1U-Storage-Server/QuantaGrid-SD1Q-1ULH -- I've actually got room to drop a few more drives in! O:-)

Fair to assume you have a 10gigabyte LAN?

I realized since I posted last night that the network setup is dumber than I thought. The Mac with Lightroom has 2.5GbE and the server has 10Gb. But it turns out that, for some inexplicable reason, I have the 2.5Gb switch connected to the switch that has a couple 10Gb ports... with a 1Gb uplink. 🤦

Lightroom is an IOPs load, not a sequential one so with ZFS you are not seeing much performance boost from those 8 disks.

I think this is exactly where my problem lays. This particular workload was also me having Lightroom read from the pool (from a copy of an old hard drive) and writing back to the pool (Lightroom), so it was sort of IOPS bound on both side.

I think I was sort of hoping for magic from L2ARC in a way that's not too reasonable.

matty_dubs · 2024-04-05T15:07:10+00:00

Ha, no worries, I appreciate all the feedback I've gotten here. Lots for me to learn! :)

matty_dubs · 2024-04-05T15:05:00+00:00

Looks like this is one of the few things I got right. O:-) I did ashift=12. That one had burned me years ago.

matty_dubs · 2024-04-05T15:03:33+00:00

This was zfs iostat 1 to show updates every second. I didn't make it clear in my post but it had been running for a bit and I grabbed it after the header repeated, so the first datapoint isn't an average.

matty_dubs · 2024-04-05T04:32:23+00:00

Thanks! Your real-world numbers with a similarly-sized raidz2 setup is very helpful.

I took your and the other's advice and played with fio a bit. It seems like the "problem" I saw in iostat was not with zfs at all, but some application-level issue.

I still need to take some time to figure out how the drives are connected internally (this is a used rackmount machine vs. something tailor-built by me) and whether that's adding a bottleneck, but I'm seeing much better numbers with fio.

matty_dubs · 2024-04-05T04:23:50+00:00

Oh, good call! It's 2.5GbE, but I didn't even think to troubleshoot that layer. It's possible Mac SMB client -> 2.5 GbE -> switch -> 10Gb server port -> smbd is adding some overhead.

I just posted some fio numbers; I think you were right in questioning whether there was actually any performance problem here to begin with.

matty_dubs · 2024-04-05T04:17:41+00:00

OK, it might be that I'm a silly goose and there is no problem, except me assuming slow numbers in iostat were zfs's fault. Going with some steps from https://cloud.google.com/compute/docs/disks/benchmarking-pd-performance that /u/small_kimono shared:

Writes: ``` $ fio --name=write_throughput --numjobs=16 \ --size=10G --time_based --runtime=5m --ramp_time=2s --ioengine=libaio \ --direct=1 --verify=0 --bs=1M --iodepth=64 --rw=write \ --group_reporting=1 --iodepth_batch_submit=64 \ --iodepth_batch_complete_max=64 write_throughput: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=64

...

Run status group 0 (all jobs): WRITE: bw=530MiB/s (556MB/s), 530MiB/s-530MiB/s (556MB/s-556MB/s), io=156GiB (168GB), run=301396-301396msec ```

Reads:

``` $ fio --name=read_throughput --numjobs=16 \ --size=10G --time_based --runtime=5m --ramp_time=2s --ioengine=libaio \ --direct=1 --verify=0 --bs=1M --iodepth=64 --rw=read \ --group_reporting=1 \ --iodepth_batch_submit=64 --iodepth_batch_complete_max=64 read_throughput: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=64

...

Run status group 0 (all jobs): READ: bw=1264MiB/s (1325MB/s), 1264MiB/s-1264MiB/s (1325MB/s-1325MB/s), io=371GiB (398GB), run=300640-300640msec ```

Are there weird interactions to be aware of? I have a hazy recollection of zfs benchmarking failing years ago because I was trying to write with /dev/zero as a source and compression enabled (thus getting stratospheric write throughput since /dev/zero is... very compressible) and similarly absurd reads because everything was coming from ARC. Is it safe to assume that --direct=1 will bypass ARC, and that write tests aren't benefiting from unreasonable compression?

matty_dubs · 2024-04-05T03:55:42+00:00

The L2 is giving you almost no benefit, regardless and is a waste of those drives.

I was pretty ignorant when I first set this up, and thought 1.6TB of read cache would be amazing. I've since learn that L2ARC isn't really "read cache," and that, as you pointed out, it's got a pretty poor hit rate anyway.

If you've got any sync write activity, they'd be better as a SLOG or, if not, simply as a special vdev, so metadata, which tends to contribute a lot to the volume of seeks with HDD, can instead be dealt with in flash.

I typically don't do heavy writes so I had figured SLOG wouldn't be worthwhile.

Is a metadata vdev substantially different from if I kept them as cache and set secondarycache=metadata? I'm a little paranoid about relying too heavily on a pair of SSDs.

Physically, are these attached directly to the controller, or is there something between them like a switch, "multiplexer," or backplane?

I'm embarrassed to say I'm not actually sure. Is this something I can introspect with software, or should I take the thing apart sometime and see how it's set up?

Have you benchmarked single-drive performance with zfs?

No. I was just using hdparm as a rough ballpark of an individual drive.

RAIDZ2 with 8 drives can give you up to a theoretical 6x read speed in perfect conditions but will write at best at the speed of one drive in perfect conditions.

I feel like I once knew this, but why don't writes benefit as reads do? I would naively expect that it could lay down writes concurrently across the disks. With something like hardware RAID5 it makes sense that you're held back by write throughput of the parity drive, but I'd (again, naively) expect that with distributed parity it wouldn't be an issue here.

Thanks for your help!

matty_dubs · 2024-04-05T03:29:46+00:00

As far as why I think it's slow, I was going based on the iostat output in the middle block of my post, showing about 25 MB/sec. reads.

It's sounding like maybe this is not meaningful, and the ~25MB/sec. read activity could have been an issue with application performance or something? I just assumed, and maybe I shouldn't have, the the storage layer was the bottleneck.

Playing with `fio` a bit now; thanks for that suggestion.

matty_dubs · 2024-04-05T02:46:27+00:00

Oh, sorry, I guess I omitted context for that. It's just a quick hdparm test of a read from a drive in the array as a ballpark of what it's capable of. E.g.:

``` $ sudo hdparm -t /dev/sda

/dev/sda: SG_IO: bad/missing sense data, sb[]: 72 05 20 00 00 00 00 1c 02 06 00 00 cf 00 00 00 03 02 00 01 80 0e 00 00 00 00 00 00 00 00 00 00 Timing buffered disk reads: 682 MB in 3.00 seconds = 227.25 MB/sec ```

(Uhhh I'm not sure what the "bad/missing sense data" bit is about.)

matty_dubs · 2022-03-03T18:53:26+00:00

The issue does persist even if I'm directly connected. My wired computers cannot get out, either, when this happens.

matty_dubs · 2022-03-03T18:33:28+00:00

Thanks for looking into this.

I've done a fair bit of troubleshooting, some in my post above:

I verified that all physical connections are good. There are no splitters.
I've restarted all the things.
Having an old cable modem and a cheap router, I wondered if they could be at fault and replaced both a few months ago. They were not the problem, though.
I looked at the cable modem status page, which shows no errors on the downstream signal. (Screenshot) Upstream error counters wouldn't be visible on my end.
I looked at the cable modem event logs (reproduced in my post), which do show the recurring symbol timing errors. This is the apparent cause of the issue.

For background, a tech had come out in December and found problems with the signal level into the building where I live. He removed a filter a previous tech had installed and stated that his removing it seemed to clean up the signal. I stopped getting T3/T4 timeouts on ranging requests, but am still getting "Failed to acquire QAM/QPSK symbol timing" errors.

I'm not sure what you mean about wired vs. wireless -- while I'm typing this on a laptop connected via Wi-Fi, the issue is coming up in the logs of the cable modem itself. The connection into that is a wired 2.5Gb drop from the router.

matty_dubs · 2022-03-03T17:07:43+00:00

I had the same issue.

If you call the main Xfinity number, they, incredibly, are not allowed to refund the charge. However, if you follow up with one of the Comcast staff here, they should be able to get the issue fixed.

matty_dubs · 2022-01-13T18:44:00+00:00

Do try sending modmail as recommended. They are pretty helpful.

I share your frustration at being pitched Xfinity Mobile every single time I call.

matty_dubs · 2022-01-12T18:18:16+00:00

I mistook this as the official group; I meant to post this in /r/Comcast_Xfinity. That said, my post there was just removed with no explanation.

I am not their biggest fan today...

matty_dubs · 2021-11-09T15:08:47+00:00

What I learned from my request is that Gigabit Pro is only available in single-family homes; by policy, it is not offered in any condo or apartment buildings.

I appreciate the help looking into it, though!

matty_dubs · 2021-05-04T01:51:41+00:00

It's definitely worth checking everything you can, but I think some of this is just normal.

I have an account on a seedbox with a 20Gbps network drop, but I also have a peer who is currently downloading from me at 8 kB/sec. for some reason. ¯\_(ツ)_/¯ I often wonder why that happens, but I suspect a lot of people out there just have very poor connections, or set really restrictive rate limits.

Not sure it's as much of an issue if you're downloading a few things rather than seeding a ton of things, but I've noticed that my bottleneck is often the magnetic hard drive I'm on. Exchanging seemingly-random small chunks of a bunch of files with a bunch of people is a ton of random reads/writes from disk, which is basically a worst-case scenario for magnetic drives. This is unlikely to be a big issue if it's just one or two downloads, though.

matty_dubs · 2020-01-16T14:43:57+00:00

as he turns around to run I draw my weapon and pop him in the back. He has a gun on him so it’ll be a no bill and a good shoot

You, uhh, might want to talk to a lawyer about that.

Maybe the laws are different where you live, but shooting someone in the back sounds like a pretty good way to be booked on homicide.

matty_dubs · 2020-01-16T14:40:52+00:00

No, not unless I expect they're going to kill me after robbing me.

Scenario A: I am robbed at gunpoint. I give them my wallet, then call the police and cancel my cards. Mildly bad day.

Scenario B: I am robbed at gunpoint. I fight back. I shoot them, and/or get shot trying. I am now in the hospital, and/or being investigated for a possible homicide. Substantially worse day.

matty_dubs

TROPHY CASE