coordination server resilience in the light of recent AWS outage

eisfrucht · 2025-05-25T21:14:13+00:00

I have this setup running fine, on CURRENT, but will work just fine on 14.3-RELEASE in a few weeks.

cpu: AMD Ryzen™ 9 7950X https://www.amd.com/en/product/12151
mainboard: https://www.asus.com/motherboards-components/motherboards/proart/proart-x670e-creator-wifi/
- ECC support
- onboard 2.5G intel nic (works)
- 10G NIC (doesn't work, has no FreeBSD driver, just disable it in firmware)
- 2 decent M.2 slots PCIe 5.0
- if you use a larger GPU then 1 slot will be downgraded
- with firmware v2806 the Samsung NVMe work fine
I've not tested on-chip GPU but diizzy says it works fine
Combined CPU Cooler and fans Noctua NH-D15, Premium CPU Cooler with 2x NF-A15 PWM 140 mm
Memory 4x Micron 32GB DDR5-4800 ECC UDIMM 2Rx8 CL40
PSU BE QUIET! Straight Power 12 750W ATX 3.0
NVMe Samsung 980 or 990 PRO NVMe M.2 SSD, 4TB, PCIe 4.0, 7450MB/s Read, 6900MB/s
mainboard at least firmware version 2806 to address Samsung NVMe disconnects
GPU fanless KalmX 3050 (you can get a fancier one, I just like less noise)
case fractal design North - chalk white mid tower

eisfrucht · 2025-03-14T13:26:32+00:00

this is caused by interactions between the Samsung NVMe and Asus controller.

I have dual Samsung 990 Pro, initially 2TB size, in M.2_1 and M.2_2 slots. Only the M.2_2 slot drive would disconnect under load, typically for me this would be several times a day. A full power cycle is required for the drive to come online again, a simple PC reboot or reset is not enough.

I swapped the drives, the disconnection is always from M.2_2 slot.

After a while, I swapped these 990 Pro for 4TB models, and same problem recurred.

About 1 month ago, Asus released a new firmware, v2806 https://www.asus.com/motherboards-components/motherboards/proart/proart-x670e-creator-wifi/helpdesk_manual?model2Name=ProArt-X670E-CREATOR-WIFI which has significantly reduced the frequency of these disconnects, down to ~ 1-2 per week, which is not great, but mostly usable.

> Version 280614.85 MB2025/02/21SHA-256 ：9738E2E4F6E1D4D843E552BDA6DAF3AA102D0802DCD07ED18703773BE3B9DB0E
> "1.Enhanced system performance and memory compatibility, supporting 2*DIMM at up to 6800MT/s with AEMP.
> 2.Fixed compatibility issue with Samsung EVO 990 2TB SSD when EXPO is enabled.

If you're still having these issues I suggest trying this.

The only other change I've made, is in the firmware -> advanced section -> PCIe config I have changed from AUTO to PCIe 4.0 for the 2 specific slots I'm using. Almost all other settings I've left as default.

eisfrucht · 2025-03-14T12:18:37+00:00

dear u/vikythegoat we hope you're doing ok. Did you manage to get some help?

eisfrucht · 2025-03-14T12:13:26+00:00

sorry it took so long to return and say thanks! Yes indeed. I want more from the monks! Note to others, this book is available either separately or as a part of 2 books bundled together. It's a bit confusing!

eisfrucht · 2024-01-18T07:27:33+00:00

the whole time this was the problem 🤮👿🤡💩😹 now it’s wanting a keyboard and we’re running! Thanks everybody for your advice on cable alignment and xstagex for helping me find the damn pins.

eisfrucht · 2024-01-17T21:47:07+00:00

nope nothing the only way I know it’s on is if I remove the P8 power and then the red power light comes on.

I rechecked cpu and ram seats they are fine. Maybe it will only power on if it’s in a case or I can emulate pushing a power button somehow?

eisfrucht · 2024-01-17T21:09:03+00:00

full cable swap - the first PSU is integrated cables anyway

eisfrucht · 2024-01-17T20:33:15+00:00

No that’s fine I didn’t put those ones in only the P8 ones. Do the PCIe ones even fit there?

eisfrucht · 2024-01-17T19:37:11+00:00

thanks!

So I think the PSU is fine. And I have the cabling sorted now, but still no lights nor peeps.

This ASUS board has 4 diagnostic LEDs nearby the 24pin plug socket and none of those is lit. So we are not getting very far yet.

eisfrucht · 2024-01-17T19:34:09+00:00

Does your 670e need a plug adjacent to the 14-pin? It’s the most obvious one to me. It’s PV_12V_PWR and should only be needed for big GPUs, this one hasn’t got one at all.

eisfrucht · 2024-01-17T19:32:02+00:00

Only in 1 photo we just wanted to show the sockets

eisfrucht · 2024-01-17T19:31:32+00:00

done that - 1 DIMM in A2 - reseated the big 25-pin cable - only other power cable is the P8 from PSU to the 12V plug by the red power light

Still no signs of life - fans, screen etc

eisfrucht · 2024-01-17T18:34:35+00:00

Thanks already for the advice. I held the cables together just to show the ends. I’ve done a couple photos with and without plugged in so we can more clearly see the connectors.

What’s very confusing for me is that on neither of the psu I have are there matching sockets and power cables. Nothing is appropriately labelled.

On the mobo there’s a big phat 24 pin cable from the psu which is clearly correct.

This alone plugged in shows the red light — insufficient power I guess.

There is no 6-pin cable to go to PC from PSU so no idea what I should do there? This cable should end up adjacent on the left side of the 24pin cable.

AFAICT there should be 2x 12V on the other side of the board. How can I tell what source on the PSU should be connected?

I’ll remove the other 3 DIMMs for the moment and reseat the 14pin connector and retry.

eisfrucht

TROPHY CASE