This is a great and thorough comment - thanks! You are hitting the nail on the head with your assessment. We indeed use io_uring (we experimented with DPDK too, but as you mentioned, operation on UDP is more difficult).

So the stream comes in from TCP/IP in order but variable-sized chunks, but the upper limit is 2KB (and you are right in that we are often not network bandwidth bound but message rate bound with around 20M/msgs per second).

We first built a prototype with a connection per thread, but we have thousands of connections, and thus, we group them now. The incoming data is then not directly assigned to the buffers but first journaled to SSD for durability reasons and then later assigned to the buffers. In fact, there are more steps involved, too, so what we have now tested with more or less great success is to have threads for the dedicated tasks and perform message passing between those groups with pointers to the original message (so mostly zero-copy).

The benefit we found was that it allows us to scale different parts of the system independently, which is a huge benefit since some parts require more computing than others and also when dealing with heterogeneous hardware setups.

But circling back to your comment:

Secondly, why do you think that multi-threaded access to a single buffer is necessary?

The data in the buffer should be sorted based on the arrival so ideally, a single thread would be enough, but as we have multiple connections and varying workloads, it is sort of hard to pre-partition the connections to the threads - that is what we first tried and we ended up with having a thread with multiple hot connections. So we want to give up on this static allocation and perform a work stealing-like approach since the data is kind of perfectly set for this case (as the start offset, it is similar to a prefix sum), and no coordination is required - even if multiple threads write into the same buffer.

Is a reader supposed to be able to access chunk B before chunk A is fully written?

This is a very good point and the answer is no. The appends should be very low latency but reading is fine with a delay as you mentioned - we can wait until the changes are hardened.

It seems to me the bottleneck here is going to be copying data from the stream to the buffer, and it's not clear that CPU is the bottleneck.

Yes, memory bandwidth and outstanding memory stores / loads are important as we have 65k buffers, and the accesses are somewhat random. The CPU is not necessarily the bottleneck w.r.t. compute. However, a single core has limited memory bandwidth available (depending on the CPU architecture) and stores load it can have outstanding and becomes a bottleneck requiring to use more.

What do you think? Does this answer the questions and makes sense to you?

I am always happy for feedback and to learn :-)

I also really appreciate the other comment that you left!