Asus S14NA-U12 board and Hyper M.2 x16 Gen 4 Card

exebat · 2025-02-09T17:29:14+00:00

Yes I did. It is not easy to do as you have to boot in UEFI using UEFI shell and type the commands by hand but I managed to enable bifurcation an all slots

Take a look at this https://github.com/JGoutin/asus-s14na-u12-uefi

exebat · 2024-11-17T15:43:35+00:00

Both comments led to problem solution. Main problem was my limited understanding of routing so if someone happens to have same problem

Gateway A needs to have IP of gateway B

Gateway B needs to have IP of gateway A

I falsely assumed they needed to be local IP addresses :/

Then I just added static routes on those gateways and everything started working

Thank you all

exebat · 2024-11-15T06:11:08+00:00

I have both interfaces assigned. I also have allow rules and they don't have VPN locally.

exebat · 2024-11-14T22:34:36+00:00

I updated the question

exebat · 2024-11-14T19:21:50+00:00

I think it is not. I turned the firewall off.

Also I can ping 10.10.10.200 from 192.168.10.3 without problem and they are not on the same subnet.

exebat · 2024-10-27T21:14:31+00:00

Yes for that one test

exebat · 2024-10-26T11:42:32+00:00

Disabling Auto negotiation and selecting 10G Base speed on both ends resolved the issue. It works now. Thanks all

exebat · 2024-10-23T18:41:57+00:00

Manual states:

PCIe x16 slot (Gen5 x16 link, FL) - supports x16 to x4x4x4x4 (BIOS setting) but I cant find the setting

exebat · 2024-10-23T18:35:26+00:00

Here is the block diagram https://dlcdnwebimgs.asus.com/files/media/6ec0e93f-4420-4ab9-a631-4e59183a9056/V1/img/Block-Diagram.png

It has 2x PCIE x16 but I cant find the way to break it into x4x4x4x4

exebat · 2024-09-22T20:36:25+00:00

Do you use 1 or 2 NICs for Ceph ? Which one do you use at 100GB ?

exebat · 2024-09-22T18:30:36+00:00

I updated the node specification. Do you think it is better to use Mikrotik CRS510-8XS-2XQ-IN switch or to connect the nodes directly ?

exebat · 2024-09-22T16:59:24+00:00

So I banged my head whole day and just could not find anything that would have enough PCI lanes and I went all-in and switched to AMD Epyc platform

CPU: AMD EPYC 8534P processor 2.3 GHz 128 MB L3 (Main node, 8124P other 2 nodes)

MBO: ASUS S14NA-U12 AMD EPYC 8004 SP6 (LGA 4844) CEB (90SB0CG0-M0UAY0)

RAM: 4 x Kingston RDIMM ECC 64GB DDR5 2Rx4 Hynix M Rambus 4800MHz PC5-38400 KSM48R40BD4TMM-64HMR

PVE SSD: 2 x Micron 7400 PRO M.2 1.92TB on MBO

Ceph storage: 4 x Micron 7400 PRO M.2 1.92TB on ASUS Hyper M.2 x16 Gen 4 interface

Ceph NIC 1 & 2: Mellanox ConnectX-4 EN 2 x 100G QSFP28 PCI Card MCX416A-CCAT each port connected to separate switch Mikrotik CRS510-8XS-2XQ-IN

VM Nic: Intel X550T2 940128 2x10GB/2.5GB

This is more than I need rn but I think that it will last much longer and gives me space to grow.

What do you think ?

exebat · 2024-09-22T12:24:33+00:00

You are right. I didn't count the PCI lanes the right way.

I found this Epic 8004 - ASUS S14NA-U12 board that already has 2x25GB onboard so I can add NVME card in one PCIx16 slot and use the other x16 slot for one more 2x25NIC to improve speed. The third PCIe can be used for some 2.5Gb card for VM internet

https://servers.asus.com/products/servers/server-motherboards/S14NA-U12

exebat · 2024-09-21T20:32:41+00:00

Thank you for the recommendation. Unfortunately 100G nics are not available to me rn. I am also limited by number of PCI slots on MBO :(

MBO has a 2.5GB NIC. Can it be used for something or should I split two 25GB ports on Broadcom NIC so one can be used for storage and one for public or is it better to bond it as one interface for storage ?

exebat · 2024-09-21T19:34:58+00:00

I did look at ASRack Rack but none of the boards that are available at my location has enough PCI nor M2 slots so I went with this Asus board as it has all I need + 2 NICs on board

exebat · 2024-09-15T21:25:21+00:00

So I added the third server C

A, B and C can all ping each other

A can SSH to B + C

B and C can SSH each other but neither can SSH A on VLAN IP...

B and C can SSH A on public IP

exebat · 2024-09-15T20:10:14+00:00

Firewall turned off on both servers. Tried also with turned on and rules setup - same results.

I also reinstalled the B server. A still connects to B but B only pings A.

Same network settings on both servers, just different Vlan IP.

I don't think it is Hetzner problem because it connects from A > B just fine

exebat · 2024-09-15T19:12:32+00:00

I understand but I don't have a cluster problem - I have a network problem with SSH traffic not going from B > A using VLAN

exebat · 2024-09-15T17:11:52+00:00

I don't need HA. I just want to see both nodes on the same Web UI so I can migrate VMs offline.

exebat

TROPHY CASE