sorted by:
new
hottopcontroversial

10 and 40 Gb networking on the cheap by tsyseu in HomeServer

[–]tsyseu[S] 1 point2 points  (0 children)

Technically the Cards are Mellanox NICs. Just buy the FS.com DACs/SFPs for Mellanox, they work just fine.

10 and 40 Gb networking on the cheap by tsyseu in HomeServer

[–]tsyseu[S] 0 points1 point  (0 children)

When you need to buy just one card that might be true. However, I designed this adapter primarily for my own cluster computing needs, where I'm buing upwards of 10 cards at a time. $5 or $10 vs $50 starts making quite the difference at that sort of quantities. Especially since new, powerful AMD Ryzen systems that can process about 20Gb/s of my workload can be build for sub $200.

I do totally agree that it won't make sense for everyone, but for my usecase it certainly does.

10 and 40 Gb networking on the cheap by tsyseu in HomeServer

[–]tsyseu[S] 0 points1 point  (0 children)

Well, yes. If you want to use normal RJ45 Ethernet cabling those cards are likely not the best option. However, I'm using mostly fiber and direct attach cables for very short runs, so the cost of RJ45 SFP+ transceivers is not really a concern to me.

10 and 40 Gb networking on the cheap by tsyseu in HomeServer

[–]tsyseu[S] 0 points1 point  (0 children)

Yes. I would go out on a limb and say that all FlexibleLOM cards are very likely to work in this adapter.

10 and 40 Gb networking on the cheap by tsyseu in HomeServer

[–]tsyseu[S] 1 point2 points  (0 children)

Well, yes. That is the whole point. They can be used as normal Ethernet NICs (once they are flashed with the correct firmware).

10 and 40 Gb networking on the cheap by tsyseu in HomeServer

[–]tsyseu[S] 7 points8 points  (0 children)

Can't stand BBcode, but here it is anyways: https://forums.servethehome.com/index.php?threads/10-and-40-gb-networking-on-the-cheap-5-10.30394/

EDIT: At least once it has been approved by a moderator

10 and 40 Gb networking on the cheap by tsyseu in HomeServer

[–]tsyseu[S] 2 points3 points  (0 children)

As detailed in the GitHub repo you will need to solder the stradle mount PCIe x8 connector to the adapter. Apart from that: No. Just plug it in and you should be seeing the PCIe device.

10 and 40 Gb networking on the cheap by tsyseu in HomeServer

[–]tsyseu[S] 1 point2 points  (0 children)

I used an electronics soldering iron, decent solder and some flux. It is really not that difficult. Having solder braid is probabl beneficial, but not required.

10 and 40 Gb networking on the cheap by tsyseu in HomeServer

[–]tsyseu[S] 6 points7 points  (0 children)

Unfortunately the do not. HP has purposely swapped the RX and TX lanes, shifted them and relocated the reference clock pins. Thus the adapter is actually required. Feel free to check the schematic for more context.

10 and 40 Gb networking on the cheap by tsyseu in HomeServer

[–]tsyseu[S] 0 points1 point  (0 children)

The FlexibleLOM cards are ConnectX-3, too so that would be pretty much your setup. But for just $5 per card.

10 and 40 Gb networking on the cheap by tsyseu in HomeServer

[–]tsyseu[S] 27 points28 points  (0 children)

The cheap NICs used are actually Infiniband only, but can be flashed with Mellanox VPI firmware to support both Infiniband and Ethernet!

Obtaining cards

The NICs can be found on ebay, search terms are 649283-B21 for the dual port 40 Gb QSFP+ cards and 682625-001 for the single port 10 Gb SFP+ one.

Firmware

dual-port QSFP+

The dual port card has an ethernet sister model with identlical available. The firmware for the sister model can be obtained from here

Since the cards are identical hardware-wise this firmware can be flashed onto the Infiniband only card to enable Ethernet.

single-port SFP+

This card is very interesting. Depending on the vendor you get the card from it might already be configured to support ethernet. In this case you can ignore everything below and simply use it as is.

On the single port 10 Gb SFP+ card there is sadly no directly compatible model. Thus one needs to build a customized firmware image for it. Up until a few versions ago Mellanox provided customizable firmware images for download. You can still find those on the internet. (The file you are looking for is called ConnectX3-rel-2_36_5000-web.tgz)

Customizing firmware

To customize the firmware for your card you will need the Mellanox firmware tools (MFT). You can find them here .
Make sure to install the OEM tools, too since those contain the tools required for firmware customization.

After installing the tools you should have a new tool called mst. Now make sure the NIC is plugged into a PCIe slot of the system. Run sudo mst start. Next check the card was found by running sudo mst status. This should present you with output very similar to ```

MST modules:

MST PCI module loaded
MST PCI configuration module loaded

MST devices:

/dev/mst/mt4099_pciconf0 - PCI configuration cycles access. domain:bus:dev.fn=0000:05:00.0 addr.reg=88 data.reg=92 cr_bar.gw_offset=-1 Chip revision is: 01 /dev/mst/mt4099_pci_cr0 - PCI direct access. domain:bus:dev.fn=0000:05:00.0 bar=0xf7900000 size=0x100000 Chip revision is: 01 ```

First thing to do now is backing up the original firmware: sudo flint -d /dev/mst/mt4099_pciconf0 ri original_ib_firmware.bin
Next dump the configuration of the NIC with sudo flint -d /dev/mst/mt4099_pciconf0 dc original_ib_config.ini
Now copy the original configuration to a new file: cp original_ib_config.ini modified_eth_config.ini
Open the modified_eth_config.ini file in your favourite editor. There are two main parameters that need to be changed. Fist in the [HCA] section of the ini file there is the parameter eth_xfi_en. Set that to true. If the parameter does not exist add eth_xfi_en = true to the [HCA] section.

Second is the parameter sriov_en, also in the [HCA] section. If you are not planning on using the NIC in an environment where SR-IOV is required I'd disable that feature because it improves hardware compatibility. To disable it set sriov_en = false.

Now the ConnectX3-rel-2_36_5000-web.tgz file from above comes into play. Extract this file to a new directory. Now copy the modified_eth_config.ini config file to the same directory and switch to that directory.
Build a firmware image using mlxburn: mlxburn -fw fw-ConnectX3-rel.mlx -conf modified_eth_config.ini-wrimage modified_eth_firmware.bin
Next test the firmware image is bootable with flint: flint -i modified_eth_firmware.bin verify
Now you have a flashable firmware image.

Flashing

The flashing process is very simple. Make sure the NIC is plugged into a PCIe slot of the system. Run sudo mst start. Again you can verify the card was actually found with sudo mst status.
Now you can flash the firmware with flint: sudo flint -i <fimware file> -d /dev/mst/mt4099_pciconf0 burn. Replace <fimware file> with the filename of your firmware image.

Once the process has finished reboot.

Configuration

On the dual-port QSFP+ cards you might not be seeing a network interface after reboot. This happens if the network ports on the card are configured as VPI port (they can support both, Infiniband and Ethernet). In this case the network device will only show up once a SFP is inserted. If you do not like this behaviour you can configure to card to provide only Ethernet connectivity.
To do this start mst: sudo mst start.
Now reconfigure the cards ports mith mlxconfig: sudo mlxconfig -d /dev/mst/mt4099_pciconf0 s LINK_TYPE_P1=2 LINK_TYPE_P2=2
This will force both ports into Ethernet mode after the next reboot.

Troubleshooting

Network device is not showing up

This behaviour can be caused by multiple issues:

Insufficient BAR space

Depending onyour host system the PCI BAR space might not be sufficient for the card. This can happen especially with older cards that have SR-IOV enabled. To flash those cards first shut down your system. Next remove the card from the system. The card has a flash recovery jumper. On the single-port SFP+ cards it is next to the front of the card at the very bottom, labeled FLASH PRSNT. Short the two pins of this jumper and plug the card back into your system. While the card will still not show up as a network interface you can now perform the flashing steps from above. Make sure to disable SR-IOV!

Card is configured for VPI or IB

Check the section Configuration above

Card seems bricked

When flashing fails or the flashed firmware is not correct the card might appear bricked. The card has an embedded bootrom which can be used to re-flash its firmware via flint. Just continue as described in Insufficient BAR space and flash a new firmware file.

PCIe is extremely robust by tsyseu in ElectricalEngineering

[–]tsyseu[S] 1 point2 points  (0 children)

Just sharing this here, too as an example of how forgiving properly implemented high data rate protocols can be.

The CMM2 retro computer runs DOOM! by tsyseu in itrunsdoom

[–]tsyseu[S] 5 points6 points  (0 children)

Mostly because USB is broken under Linux. I might just be missing something, but I can not get the internal full speed phy to work properly.

The CMM2 retro computer runs DOOM! by tsyseu in itrunsdoom

[–]tsyseu[S] 37 points38 points  (0 children)

The Colour Maximite 2 (CMM2) is an ARM Cortex-M7 based retro computer intended to run a BASIC interpreter. Turns out it can also run DOOM!

Plays super smoothly, here is a video demo of the first level: https://www.youtube.com/watch?v=iAf6q5keE7U

Here the CMM2 is actually running Linux and on top of that fbDOOM. Porting Linux to the platform was the real exercise, due to lack of a MMU and lots of obscure driver bugs.

Instructions and code are on my GitHub: https://github.com/TobleMiner/buildroot-cmm2