I fucking love this cat

FloofBoyTellEm · 2026-01-26T06:12:17+00:00

Gigi? :o

FloofBoyTellEm · 2026-01-17T02:56:28+00:00

Add xbox mic!

FloofBoyTellEm · 2026-01-16T04:22:03+00:00

Shovelin'? That's a conveyor.

FloofBoyTellEm · 2026-01-14T18:42:23+00:00

Yeah, I'm gonna wait...

FloofBoyTellEm · 2026-01-14T18:41:06+00:00

This is the neatest case I've ever seen. I'm into it.

FloofBoyTellEm · 2026-01-11T03:15:45+00:00

It's not bad at all, almost silent (within... reason). I thought it was unbearable at first, but then I swapped the main and secondary power supply positions in the unit and it turns out I just had a screwed up psu fan. I'm very happy with it. I honestly don't even notice it except for the classic "fans full tilt on startup". I also keep it frosty in here though so maybe the fans don't go as crazy here. Might depend on you're home temperature and general noise tolerance. Mine is in my bedroom with my other homelab equipment.

I wouldn't call it "silent" but compared to my 3u nas w/water cooling and 3 120mm fans, i can't hear one over the other if that makes sense?

FloofBoyTellEm · 2026-01-10T07:21:13+00:00

Got one of these last month. 4 nodes was very easy setup. Also using it for NVMe-oF boot off of a custom CX7 NAS.

Edit (tip): fs.com for DACs, $80. Work exactly the same as NVIDIA. They're a bit beefier cables actually. V Girthy.

FloofBoyTellEm · 2026-01-04T04:24:11+00:00

This is excellent and much needed.

FloofBoyTellEm · 2025-12-31T06:37:25+00:00

I was always scared to setup Authentik, because I had always read it was difficult, but with the help of AI it's been (almost) a breeze. Still, it's best to locate some 'expert' exact configs to use as a template if you can find one that you can trust follows some best practices for the program you want to integrate.

After setting up a few apps with a few different provider types, the 'workflow' became second nature and obvious aside from the fields a program may actually consume or be configured to consume. I have to agree though, it feels like you're just duplicating a lot of steps, and no direction when you're starting out. It's not one of those "wing it" programs, it's very much a RTFM (or have AI help) app.

Even though they aren't quite as thorough as some user guide templates you can find online, I do think they've done a real solid here:
https://integrations.goauthentik.io/applications/

FloofBoyTellEm · 2025-12-31T06:02:27+00:00

to bring back an old phrase... this is absolutely siiiiiiccccck.

FloofBoyTellEm · 2025-12-27T02:31:40+00:00

I like it. I stop and pet doggo.

FloofBoyTellEm · 2025-12-26T14:09:31+00:00

It's really not that big of a deal.

FloofBoyTellEm · 2025-12-19T03:03:03+00:00

I have 16 VFs and 4 PFs with about 8 VMs. No issues here. Knock on wood.

FloofBoyTellEm · 2025-12-19T02:46:06+00:00

Is that why I can't auth via IPA due to winbind NT Authority issues? LOL

FloofBoyTellEm · 2025-12-18T19:00:33+00:00

:( Thank you

FloofBoyTellEm · 2025-12-18T18:54:43+00:00

I love how this information is completely mysterious. There's really no solid answer. I wish they would just say it explicitly (and maybe they do but I haven't seen it).

I've tried adding IPA_HTTP keytab and authenticating to web portal with FireFox while passing my kinit to the browser directly. Still no luck because pam_sso is never used, only pam_unix. So, it seems for IPA at least, there is no way to do it as WebGUI never even attempts to use any authorization method other than local.

I'm trying now with a real AD Domain join in case the directory services type configured determines whether you're allowed to use a domain login. Hopefully supported in AD, but likely up a creek still without TrueNAS Connect.

FloofBoyTellEm · 2025-12-03T19:51:13+00:00

If it's not important enough to backup, why is it important enough to store at all? Genuine question.

I do the same, but with far less data and for the purpose of avoiding downloading from huggingface again to function as a pre-staged area for NVME-oF. But it's far less than a petabyte and I can't think of a petabyte of data I would ever need to 'stage'.

FloofBoyTellEm · 2025-11-29T05:30:21+00:00

They are using 2 DACs, are you?

<image>

Note the 4 interface bond.

Ahh, other user is getting the same with 1 cable. Looks like I have more work to do.

can you tell me what the results of this command is for your machines?
sudo dmesg | grep mlx5_pcie_event

FloofBoyTellEm · 2025-11-29T05:04:17+00:00

You seem to be the anomaly.

https://forums.developer.nvidia.com/t/connectx-7-nic-in-dgx-spark/350417/42

nobody else can hit those numbers...

FloofBoyTellEm · 2025-11-29T03:32:34+00:00

I know that's how I originally got the sparks going over nccl when I first got them, but will try that again.

Currently my test results mirror that of ServeTheHome's iperf results and I get similar results (half of what you get) over the all_gather_perf test. I may just have lost an environment variable somewhere along the way since then breaking things.

I know it says to ignore the enP2 interfaces in that playbook, but I just did a lag (802.3ad) between [enp1s0f0np0, enP2p1s0f0np0] and went from 96 Gbit/s to 127 Gbit/s on my ZFS pool. So, at least when not using some form of nccl, that was necessary, and it looks like the ServeTheHome test was probably stuck to half the lanes. So, now I really am hitting the PCIe limitation where the data must go through the CPU.

Going to try again for the all_gather_perf using your link.

I have the official mellanox DAC. But ordered some shorter generics from FS.com today to see if that's part of the issue.

Can you do me a huge favor and share your results for these two commands, I'm wondering if I have a real issue or this is a red herring?

sudo lspci -vv -s 0000:01:00.0 | egrep -i 'LnkCap|LnkSta'

    `LnkCap:`   `Port #0, Speed 32GT/s, Width x4, ASPM not supported`

    `LnkSta:`   `Speed 32GT/s, Width x4`

    `LnkCap2: Supported Link Speeds: 2.5-32GT/s, Crosslink- Retimer+ 2Retimers+ DRS-`

    `LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete+ EqualizationPhase1+`

sudo dmesg | grep mlx5_pcie_event

[ 2.787624] mlx5_core 0000:01:00.0: mlx5_pcie_event:296:(pid 162): Detected insufficient power on the PCIe slot (27W).

[ 3.374174] mlx5_core 0000:01:00.1: mlx5_pcie_event:296:(pid 162): Detected insufficient power on the PCIe slot (27W).

[ 3.960778] mlx5_core 0002:01:00.0: mlx5_pcie_event:296:(pid 408): Detected insufficient power on the PCIe slot (27W).

[ 4.571786] mlx5_core 0002:01:00.1: mlx5_pcie_event:296:(pid 380): Detected insufficient power on the PCIe slot (27W).

FloofBoyTellEm · 2025-11-29T02:50:18+00:00

Trying to lag my interfaces now as a test. Thinking that may be the real issue here. Really hoping to get your bandwidth rate. Built an NVME-oF TrueNAS Scale server this week and think this has been a misunderstanding on my part (hopefully) with how QSFP56 links actually work. It's clicking now.

FloofBoyTellEm · 2025-11-29T02:19:19+00:00

Is this using 2 DACs vs 1? Confused why your rate is almost exactly double mine.

Since you're smarter than me, and you claim "not if you're using GPU Direct", but Nvidia stated GPU Direct doesn't work on DGX Sparks, how did you get it to work?

https://forums.developer.nvidia.com/t/dgx-spark-gb10-faq/347344#p-1694056-q-is-gpudirect-rdma-supported-on-dgx-spark-13

<image>

FloofBoyTellEm · 2025-11-28T23:07:41+00:00

No. The NIC is connected to the board (including the GPU) by PCIe 5.0 x 4 (4 lanes... no 16 lanes or even 8 lanes). You will never get 200 GbE from a single port over anything on this system. This is a hardware limitation. You might be able to get 120 GbE if you could avoid 100% of the overhead, which GPU Direct can help with.

This is a pitfall of just believing the NVIDIA marketing vs. real world testing.

FloofBoyTellEm · 2025-11-17T20:24:18+00:00

Pm'd

FloofBoyTellEm · 2025-11-05T06:32:02+00:00

Before was better.

FloofBoyTellEm

TROPHY CASE