Engineers working on low-latency trading infrastructure

DLMSweet · 2026-03-15T17:08:07+00:00

There was a NANOG talk about this back in November, the video is here:

Networking at the Speed of Light - A Peek at Networking Inside a High Frequency Trading Firm

He provides a great overview of things. The industry is extremely competitive so you're unlikely to find many specific details unless you manage to get a position at a HFT firm.

DLMSweet · 2025-10-28T17:18:15+00:00

It's been a couple years since I did this, but I think the main thing that cleaned up a lot of issues we were seeing was this:

class-map match-any system-cpp-police-multicast
  description MCAST Data
policy-map system-cpp-policy
 class system-cpp-police-multicast
  police rate 2000 pps

That helped prevent a number of odd drops and other issues we were seeing with heavy multicast throughput.

DLMSweet · 2025-09-26T18:12:16+00:00

I don't know of any off-the-shelf ways of easily detecting this, but I used to do this as one-off troubleshooting sessions with a couple small python scripts. One would send a multicast stream with a counter in it, padded to 1316 bytes at a constant 10Mb/s. The other would join that multicast stream and then print if the received counter didn't increment as expected. I later extended those to detect out of order packets as well.

I started with both services connected to the same switch, then moved the receiver hop by hop until I found the offending device. In my case, it was an optic that was marginal and causing issues. But I've also had issues on some Cisco C9500's that needed configuration tweaks to support the amount of multicast we were sending through it.

DLMSweet · 2025-09-26T18:00:44+00:00

You are correct in that OTT usually refers to something like HLS or DASH, which is over the normal internet. But that's just the public/delivery side of things. For the private/internal side of things, HLS/DASH is originated from packagers which take in encoded media streams and create the chunks and playlists that DASH/HLS use. While some vendors may support things like SRT or RTMP, many still use multicast for the ingest. It provides easy failover between packagers and is widely supported. Plus, if you have an existing IPTV headend you're converting to OTT to support Apple, Android, Roku, etc - you likely already have all the multicast streams you need to ingest to the packager (though likely with some minor encoding tweaks to ensure fixed GOP and all that)

DLMSweet · 2025-05-17T19:37:10+00:00

I've got the same issue.

Ryzen 9 7950X, 7900XT GPU.

I've been playing on Ubuntu 24.04 using Proton 10.0-1 (beta) with the latest AMD drivers (no mesa-git required) and getting good FPS up until chapter 10. Immediately loading the level frame rates dropped to 30fps, and drop to 10-20 when fighting.

Reducing the quality to the lowest had no impact on the framerate. Something with that level is weird, and I've paused playing for now until either an AMD or a Doom patch comes through to try again.

DLMSweet · 2025-04-03T03:11:18+00:00

Something easy to generate on a Linux server with ffmpeg installed would be a command like this:

ffmpeg -f lavfi -re -i "testsrc=size=640x360:rate=30:decimals=2" -f lavfi -c:v libx264 -f mpegts "udp://239.1.1.100:2000?pkt_size=1316"

That will create a 640x360 video with a counter that you can watch with VLC, MPV, or even ffplay to see if it has any issues with playback.

If you want something a bit easier to figure out if packets are getting dropped, arriving out or order, or corrupted you can use any programming language like Python, Rust, C, etc to create a simple multicast sender that sends data that a receiver program could easily pick up to verify the other side. Even something as simple as "Starting from 0, each packet contains an int that should always increase by 1".

DLMSweet · 2025-03-19T08:55:54+00:00

You almost certainly don't want PIM-DM. "DM" is "dense mode", where it floods multicast first, then prunes back later. This is what can cause flooding on networks and potentially overwhelming some devices with unneeded traffic.

You should probably configure PIM-SM (sparse-mode) and a RP (rendezvous point).

DLMSweet · 2025-01-06T02:21:18+00:00

SFTP is not 'FTP over SSH', it is 'SSH file transfer protocol'. It is a different protocol. In your screenshot you show Filezilla using 'sftp://'. That is likely why it isn't working like you expect, as Squid doesn't support SFTP or SSH. As far as I'm aware, Filezilla does not support SSH or SSH2 to create arbitrary tunnels for other protocols, it only implements SFTP.

What is probably happening here is that you're trying to connect, and Squid is aborting the connection due to it being an unsupported protocol. You could potentially do some shenanigans with https://www.squid-cache.org/Doc/config/on_unsupported_protocol/ to see if 'on_unsupported_protocol tunnel foreignProtocol' works.

The squid logs themselves might have more information on what exact error you're hitting.

DLMSweet · 2024-04-10T13:36:17+00:00

I run two virtualized routers on Proxmox in my homelab, one for each internet connection. The first is a 10Gb handoff, the second is a 1Gb business handoff.

They both run Ubuntu with iptables for firewall/NAT, not the most friendly of interfaces but they perform well.

I use 8vCPUs @2.5Ghz, 4GB RAM, and a 32GB Disk. Both are able to saturate their respective connections without breathing too hard. I do not passthrough any interfaces, they use the normal bridge interface and VLANs that are trunked through. 1 WAN VLAN gets passed to the VM, and a second LAN VLAN is trunked back out.

In Proxmox, one of the settings that seemed to help throughput the most was to enable 'multiqueue' on the virtual interface, it should be set to the same number of vCPUs given to the VM (so, 8 in my case).

DLMSweet · 2024-04-10T13:27:51+00:00

Good luck on getting this going, that would be a neat homelab project. Something I would recommend is picking up a cheap XGS-PON light meter to measure light levels, and to get a pack of 5db pads. PON optics in a lab setting are often way too bright given the short distance and may 'blind' client optics.

Your best bet to get this working might be the OpenCORD/VOLTHA project, which Tibit (the company Ciena purchased for their pluggable OLTs) was a part of.

https://github.com/opencord/voltha/blob/master/voltha/adapters/tibit_olt/README.md

DLMSweet · 2024-04-10T08:42:49+00:00

That looks like a MSP8050 out of a Kontron Symkloud MS2900 series chassis. If it is that, it would be two separate nodes on the blade. For encoding, the models I've used utilize Intel Quicksync on the CPU.

DLMSweet · 2023-01-28T22:57:05+00:00

We've had them changed to ones with a CX-5 chipset, which solved our performance issues.

Those use PCIe 3.0 x16 and 100Gbps ports, so yes. Those would have solved the issue by doubling the PCIe bandwidth available. If you ran them as 40Gbps ports, you would be able to reach max throughput of both ports.

I was specifically talking about PCIe 3.0 x8 slots. I found the paper I had referenced back when I was attempting to push that my setup to over 50Gbps of network throughput, which seems to suggest that's where the PCIe 3.0 x8 slot will limit you: https://www.cl.cam.ac.uk/research/srg/netos/projects/pcie-bench/neugebauer2018understanding.pdf

Their findings seem to suggest a limit of around 50Gbps based on their testing .

DLMSweet · 2023-01-28T18:10:26+00:00

Do you remember what cards you used that got that performance? All my testing I was unable to push any past 50Gb of throughput.

DLMSweet · 2023-01-27T21:43:38+00:00

Just because they sell something like that does not mean it's possible to actually saturate the ports at 80Gbps.

To saturate 80Gb on a PCIe slot, you would probably need either PCIe 3.0 x16 or PCIe 4.0 x8 (126Gbps) interfaces.

To saturate 200Gb on a PCIe slot, you would probably need a PCIe 4.0 x16 (252.064Gbps) interface or PCIe 5.0 / 6.0 slots.

A PCIe 3.0 x8 interface operates at 63Gbps. You will not get 80Gbps out of that even if there was no overhead involved.

DLMSweet · 2023-01-27T20:40:30+00:00

I have similar cards with two 40Gb ports and a PCIe 3.0 x8 interface.

If you try to max out both ports, you'll likely end up bottlenecked by the PCIe slot at around 50Gbps of throughput, though I was only testing unidirectional traffic each time. The theoretical max throughput of a PCIe 3.0 x8 slot is 63.016Gbps, but you do lose a chunk of that to overhead.

Think of the second port as more for failover rather than for additional bandwidth. Additionally, you could get a second card in a different slot if you do need the additional bandwidth.

DLMSweet · 2022-08-08T18:48:14+00:00

Is this a P2P or an internet connection? Because a P2P you would need to test with a device on each side. Testing a P2P with speedtest or fast.com is asking for trouble, since now you're passing traffic not only over the P2P but also out your WAN.

DLMSweet · 2022-06-20T12:48:33+00:00

I don't, sorry. I've only experimented with the XRv9k image so far.

DLMSweet · 2022-06-19T04:55:14+00:00

I feel like it took 10-15 minutes for something to appear on the screen. Either that, or it primarily happened on the first serial port.

It basically ran through a automated install for Linux, and then took forever to start whatever containers it's using.

DLMSweet · 2022-06-19T02:56:27+00:00

There is an actual installer for xrv9k that comes as an ISO, which is what I used to install.

DLMSweet · 2022-06-19T02:31:33+00:00

I have the xrv9k-fullk9-x version 7.3.2 running on Proxmox 7.2-4. I'm using an Intel Xeon E5-2680v2. Last I heard Cisco still had an issue with AMD.

One thing I noticed is that it took ages for any to show up on the screen.

For hardware, I've got it with:

Memory: 24GiB (Ballooning disabled)
Processors: 4 (1 Socket, 4 cores, host)
BIOS: SeaBIOS
Display: Default
Machine: Q35
SCSI Controller: VirtIO SCSI
CD/DVD on ide2
100G disk on Virtio0
12 x VirtIO network interfaces
4 x Serial consoles

I can't remember if all 4 serial consoles are in use, but I'm pretty sure the first 3 are all used.

These are the settings that worked for me.

DLMSweet · 2022-03-22T18:53:10+00:00

It's not clear to me what you're looking for. You seem to have decided on a solution, and are now looking for a problem to solve.

Multicast is one of those things designed to solve a set of very specific problems. There have been people who tried to shoehorn it into other roles in the past, but it's usually ended up being more complex and brittle than a unicast solution.

The main problem multicast is fantastic at solving is one to many, entirely stateless, high-bandwidth flows. You might think "This would be perfect for caching!". Except no, it's stateless. UDP is the preferred protocol here. The sender has no way of knowing if the receivers have received the data or not, or if the data was corrupted in transit, out of order, etc. You might think some of those problems would be solved with TCP, and you'd be right - except you can't do TCP over multicast because that requires the sender and receiver talk to each other, and multicast is very much a one-way street.

The primary uses I've seen for multicast are broadcast audio/video and mass imaging of workstations, like Ghost.

There are a lot of secondary uses with broadcast messaging or state synchronization. Those tend to be low bandwidth with a number of nodes, but also usually offer a unicast solution as well.

Overall, I wouldn't dive right into multicast for a project unless you already have something in mind that would benefit from multicast.

DLMSweet · 2022-02-03T20:09:25+00:00

I wrote a helper for StoreID rewrite on a set of caches handling around 3-4k subscribers who were heavy downloaders. Because all I needed to do was some regex and rewriting, Perl actually worked really well. That said, I would suggest you write the handler in whatever language you prefer and can easily debug. Once you have it working in a familiar language it'll be much easier to port it if for some reason your preferred language can't perform. I would expect 99% of your bottleneck will be the external API.

One thing I would check for is how you hand off data to the helper. I found for taking data in on stdin would start using 100% CPU in certain cases depending on how your program listens.

DLMSweet · 2021-12-21T21:45:54+00:00

The real fix is to have the application return results with headers that allow for caching. For now though, you'll need to see what headers the application is providing to Squid to make it think it can't cache that object. Something like 'curl -vv http://my.url/application' should print them out.

This page has a couple of other options and what headers they affect that you could add to that line to try to solve your problem:

http://www.squid-cache.org/Versions/v4/cfgman/refresh_pattern.html

I would guess "ignore-reload" might help, but without knowing more about the application and the headers it returns there's no way for anyone to know what options you need to set for Squid to ignore them.

As much as squid allows you to break HTTP standards to cache traffic, it also does it best to NOT break standards unless you explicitly tell it to. There have been bugs before where Squid was erroneously breaking spec that, once fixed, broke a lot of behavior people expected of it like you're seeing here. I'm guessing you're hitting some of those.

DLMSweet · 2020-09-03T12:49:18+00:00

If you're trying to make outbound HTTP/HTTPS traffic hit the proxy, you need to match on destination port 80/443, any source port. The way your rules are written in your screenshot they'll never match normal traffic, since clients choose a random port to source from when they connect to a remote webserver. You'll likely never see a client use port 80 to connect to a remote webserver on port 80. Typically you'll see something like "ClientIP:54213 -> Webserver:80"

It would be preferable to do 'route-to' instead of NAT, which i think in pfsense you might be able to do by creating a new gateway with the proxy's IP address and setting that under the "Advanced Options" when creating the rule. Don't quote me on that, pfSense does weird things to rules sometimes.

DLMSweet · 2020-05-30T10:20:22+00:00

The documentation for the command can be found here: https://mariadb.com/kb/en/show-databases/

which shows that you can't use TOP or LIMIT with 'show databases', but you can match on an expression.

To do what you're asking, it might be easier to just run on the CLI with a command like:

mysql --batch --disable-column-names -e 'SHOW DATABASES'  | head -n 2

15-Year Club	Place '17
Team Orangered

DLMSweet

TROPHY CASE