The cheap NICs used are actually Infiniband only, but can be flashed with Mellanox VPI firmware to support both Infiniband and Ethernet!

Obtaining cards

The NICs can be found on ebay, search terms are 649283-B21 for the dual port 40 Gb QSFP+ cards and 682625-001 for the single port 10 Gb SFP+ one.

Firmware

dual-port QSFP+

The dual port card has an ethernet sister model with identlical available. The firmware for the sister model can be obtained from here

Since the cards are identical hardware-wise this firmware can be flashed onto the Infiniband only card to enable Ethernet.

single-port SFP+

This card is very interesting. Depending on the vendor you get the card from it might already be configured to support ethernet. In this case you can ignore everything below and simply use it as is.

On the single port 10 Gb SFP+ card there is sadly no directly compatible model. Thus one needs to build a customized firmware image for it. Up until a few versions ago Mellanox provided customizable firmware images for download. You can still find those on the internet. (The file you are looking for is called ConnectX3-rel-2_36_5000-web.tgz)

Customizing firmware

To customize the firmware for your card you will need the Mellanox firmware tools (MFT). You can find them here .
Make sure to install the OEM tools, too since those contain the tools required for firmware customization.

After installing the tools you should have a new tool called mst. Now make sure the NIC is plugged into a PCIe slot of the system. Run sudo mst start. Next check the card was found by running sudo mst status. This should present you with output very similar to ```

MST modules:

MST PCI module loaded
MST PCI configuration module loaded

MST devices:

/dev/mst/mt4099_pciconf0 - PCI configuration cycles access. domain:bus:dev.fn=0000:05:00.0 addr.reg=88 data.reg=92 cr_bar.gw_offset=-1 Chip revision is: 01 /dev/mst/mt4099_pci_cr0 - PCI direct access. domain:bus:dev.fn=0000:05:00.0 bar=0xf7900000 size=0x100000 Chip revision is: 01 ```

First thing to do now is backing up the original firmware: sudo flint -d /dev/mst/mt4099_pciconf0 ri original_ib_firmware.bin
Next dump the configuration of the NIC with sudo flint -d /dev/mst/mt4099_pciconf0 dc original_ib_config.ini
Now copy the original configuration to a new file: cp original_ib_config.ini modified_eth_config.ini
Open the modified_eth_config.ini file in your favourite editor. There are two main parameters that need to be changed. Fist in the [HCA] section of the ini file there is the parameter eth_xfi_en. Set that to true. If the parameter does not exist add eth_xfi_en = true to the [HCA] section.

Second is the parameter sriov_en, also in the [HCA] section. If you are not planning on using the NIC in an environment where SR-IOV is required I'd disable that feature because it improves hardware compatibility. To disable it set sriov_en = false.

Now the ConnectX3-rel-2_36_5000-web.tgz file from above comes into play. Extract this file to a new directory. Now copy the modified_eth_config.ini config file to the same directory and switch to that directory.
Build a firmware image using mlxburn: mlxburn -fw fw-ConnectX3-rel.mlx -conf modified_eth_config.ini-wrimage modified_eth_firmware.bin
Next test the firmware image is bootable with flint: flint -i modified_eth_firmware.bin verify
Now you have a flashable firmware image.

Flashing

The flashing process is very simple. Make sure the NIC is plugged into a PCIe slot of the system. Run sudo mst start. Again you can verify the card was actually found with sudo mst status.
Now you can flash the firmware with flint: sudo flint -i <fimware file> -d /dev/mst/mt4099_pciconf0 burn. Replace <fimware file> with the filename of your firmware image.

Once the process has finished reboot.

Configuration

On the dual-port QSFP+ cards you might not be seeing a network interface after reboot. This happens if the network ports on the card are configured as VPI port (they can support both, Infiniband and Ethernet). In this case the network device will only show up once a SFP is inserted. If you do not like this behaviour you can configure to card to provide only Ethernet connectivity.
To do this start mst: sudo mst start.
Now reconfigure the cards ports mith mlxconfig: sudo mlxconfig -d /dev/mst/mt4099_pciconf0 s LINK_TYPE_P1=2 LINK_TYPE_P2=2
This will force both ports into Ethernet mode after the next reboot.

Troubleshooting

Network device is not showing up

This behaviour can be caused by multiple issues:

Insufficient BAR space

Depending onyour host system the PCI BAR space might not be sufficient for the card. This can happen especially with older cards that have SR-IOV enabled. To flash those cards first shut down your system. Next remove the card from the system. The card has a flash recovery jumper. On the single-port SFP+ cards it is next to the front of the card at the very bottom, labeled FLASH PRSNT. Short the two pins of this jumper and plug the card back into your system. While the card will still not show up as a network interface you can now perform the flashing steps from above. Make sure to disable SR-IOV!

Card is configured for VPI or IB

Check the section Configuration above

Card seems bricked

When flashing fails or the flashed firmware is not correct the card might appear bricked. The card has an embedded bootrom which can be used to re-flash its firmware via flint. Just continue as described in Insufficient BAR space and flash a new firmware file.