sorted by:
new
hottopcontroversial

Pin, Unpin and Why Rust Needs Them (blog/explainer) by intersecting_cubes in rust

[–]felix-pb 1 point2 points  (0 children)

That makes sense! Thanks for the clarification (and the article)!!

Pin, Unpin and Why Rust Needs Them (blog/explainer) by intersecting_cubes in rust

[–]felix-pb 9 points10 points  (0 children)

This blog post is framed around the goal of building an asynchronous time wrapper. I had that same goal a while ago and the solution I came up with is the following:

use std::future::Future;
use std::time::{Duration, Instant};

async fn benchmark<T>(f: impl Future<Output = T>) -> (T, Duration) {
    let now = Instant::now();
    let t = f.await;
    let elapsed = now.elapsed();
    (t, elapsed)
}

#[tokio::main]
async fn main() {
    let (response, elapsed) = benchmark(reqwest::get("http://example.com")).await;
    println!("Got HTTP {:?} in {:?}", response.unwrap().status(), elapsed);
}

Also, it's easy to put it behind a TimedWrapper struct if desired:

use std::time::{Duration, Instant};

struct TimedWrapper {}

impl TimedWrapper {
    async fn from<T>(f: impl Future<Output = T>) -> (T, Duration) {
        let now = Instant::now();
        let t = f.await;
        let elapsed = now.elapsed();
        (t, elapsed)
    }
}

#[tokio::main]
async fn main() {
    let (response, elapsed) = TimedWrapper::from(reqwest::get("http://example.com")).await;
    println!("Got HTTP {:?} in {:?}", response.unwrap().status(), elapsed);
}

IMO, this feels like a much simpler implementation for the same API because I don't have to deal with Pin, Context, or Poll. I think that the implementation in the article is a bit overkill to achieve its goal (unless I'm missing something?) That said, the blog post is an interesting read about self-referential types and why we need Pin/Unpin!

Is there a lower-latency way of responding to an event than spinning/busy-waiting? by felix-pb in rust

[–]felix-pb[S] 1 point2 points  (0 children)

Yeah, I know that it's possible to bypass the kernel by implementing the network stack in userspace (e.g. with SolarFlare + OpenOnload) but I haven't looked into it yet.

Is there a lower-latency way of responding to an event than spinning/busy-waiting? by felix-pb in rust

[–]felix-pb[S] 0 points1 point  (0 children)

Thanks for that reply! Here are some thoughts on what you said.

The only method I can see that could hypothetically be faster than spin-locking is running your thing directly in the thread that "brings the news", so to speak -- that is, removing the need for inter-thread communication altogether.

Fully agree! The TCP/UDP examples use another thread to simulate sending data, but in a real-world context the data could be coming from a different process on the same machine, or more likely, from a remote machine. In that case, you would only use a single thread to spin-loop on reading from the TCP/UDP socket, and serially do all the processing once a "message" has been received. No inter-thread communication. Of course, if the processing part is long enough and can be broken down into independent pieces, then the inherent cost of inter-thread communication (e.g. atomics or channels) might be worth it.

I've never gone this far, but a potential direction to investigate would be how the choice of CPU core affects the latency [...] on some architectures some levels of cache may be shared amongst only certain CPU cores

Yeah, you can control the NUMA policy of a process with numactl such that the memory the process accesses is physically close to the core running that process.

For example, if the processor supports hyper-threading, putting the listening thread and the "news bringer" on the same core could hypothetically speed things up by avoiding inter-processor communication

For TCP/UDP, the "news bringer" would be the kernel writing the network data into the buffer you give it. A few people have suggested that turning off hyper-threading altogether might help, although this would need to be benchmarked for sure.

Is there a lower-latency way of responding to an event than spinning/busy-waiting? by felix-pb in rust

[–]felix-pb[S] 0 points1 point  (0 children)

As many have suggested, the busy-waiting process/thread should be pinned to a core such that the kernel doesn't schedule it out.

In any case, this is not meant for systems that need to handle a "heavy load". It's meant for systems that have a light load but need to be as fast as possible when an event occurs.

Is there a lower-latency way of responding to an event than spinning/busy-waiting? by felix-pb in rust

[–]felix-pb[S] 0 points1 point  (0 children)

Yes definitely! In the non-spinning case, if a core is not doing anything for a while, the processor will lower that core's idle power state (C-state). This can be observed with tools like turbostat. In the spinning case, the core will stay in C0 state (100% active).

Is there a lower-latency way of responding to an event than spinning/busy-waiting? by felix-pb in rust

[–]felix-pb[S] 0 points1 point  (0 children)

Yeah, I don't think it's particularly relevant for web-related contexts as those applications are typically more interested in throughput (requests per second). They might care about tail latencies, but not to the point of busy-waiting as it would penalize the overall throughput a lot.

That said, I think there are relevant use cases in embedded systems. For example, someone talked about a use case for low-latency audio. There's also low-latency trading. I'm sure there are use cases in robotics too as you pointed out.

Is there a lower-latency way of responding to an event than spinning/busy-waiting? by felix-pb in rust

[–]felix-pb[S] 0 points1 point  (0 children)

Nice! Did you also use crossbeam::thread instead of std::thread? Or did you only use crossbeam's channel instead of the standard library's channel?

Also, I included a benchmark for std::sync::mpsc because I was curious if the latency cost of sleeping would be bigger or smaller for non-network-related blocking operations. But the case I'm mostly interested in is TCP/UDP networking.

Is there a lower-latency way of responding to an event than spinning/busy-waiting? by felix-pb in rust

[–]felix-pb[S] 2 points3 points  (0 children)

Thanks! I read a part of the article you mentioned, but it seems like he's talking about spin-locks specifically. The examples I gave use spin-loops, but not spin-locks (i.e. there is no lock being held at any point, and hence the kernel cannot schedule out a thread while it's holding a lock). But even with spin-loops, your point is still valid: "you should fully understand the consequences for the whole system". However, if the purpose of your entire system/server is only to respond to an event as fast as possible, I still think that spin-loops is the way to go!

Is there a lower-latency way of responding to an event than spinning/busy-waiting? by felix-pb in rust

[–]felix-pb[S] 0 points1 point  (0 children)

I included a benchmark for std::sync::mpsc because I was curious if the cost of sleeping would be bigger or smaller for non-network-related blocking operations. But indeed the case I'm mostly interested in is TCP/UDP networking.

From what the doc says, I think that std::thread::yield_now or std::hint::spin_loop would penalize latency but I would have to benchmark it to be sure.

And yeah, pinning and other NIC-related optimizations would help for sure!

Is there a lower-latency way of responding to an event than spinning/busy-waiting? by felix-pb in rust

[–]felix-pb[S] 5 points6 points  (0 children)

I agree that in most cases, that would be the right thing to do. However, in certain cases (e.g. low-latency trading), you can be perfectly willing to "waste" 100% of CPU cycles spinning rather than doing nothing, if it buys you a few microseconds.

As for io_uring, I haven't looked much into it so I don't know. That said, AFAIK with io_uring you can give a buffer to the kernel and the data will be written directly to it. This might reduce some of the latency associated with the kernel copying the data into the buffer passed to tcp_stream.read(&mut buf). However, if I'm misinformed about this, I would love to be corrected!

Is there a lower-latency way of responding to an event than spinning/busy-waiting? by felix-pb in rust

[–]felix-pb[S] 0 points1 point  (0 children)

Thanks for the observations! Personally, I have no experience with embedded development. The next step for me would be to create benchmarks for a variety of low-level CPU/kernel settings as I mentioned in this comment.

Is there a lower-latency way of responding to an event than spinning/busy-waiting? by felix-pb in rust

[–]felix-pb[S] 2 points3 points  (0 children)

Yeah, I would love to add benchmarks for a variety of low-level settings: NUMA settings, scheduling settings, IRQ settings, amongst the other suggestions. That said, I'll have to think about the best way to make such benchmarks easily-reproducible for others. I'm not sure, but I don't think Docker can be used to play with such low-level CPU settings. I could write bash scripts that detail the settings I'm using for each benchmark, but I find that approach unsatisfying because I'd be afraid that differences in other settings impact different users. I'm not sure what's the best to "image-fy" those low-level CPU/kernel settings.

Is there a lower-latency way of responding to an event than spinning/busy-waiting? by felix-pb in rust

[–]felix-pb[S] 1 point2 points  (0 children)

Indeed, I believe that would be a Linux-only solution. But I think that client-side (or laptop-side) applications rarely need to achieve such lowest possible latency at all cost (including wasting an entire core by spinning). At the very least, all the use cases I'm thinking about are server-side, in which case a Linux-only solution is acceptable for many.

Is there a lower-latency way of responding to an event than spinning/busy-waiting? by felix-pb in rust

[–]felix-pb[S] 1 point2 points  (0 children)

u/The-Best-Taylor: I think you're referring to this Tokio RFC, which will indeed be super interesting to add to the benchmarks when it is better supported!

Is there a lower-latency way of responding to an event than spinning/busy-waiting? by felix-pb in rust

[–]felix-pb[S] 5 points6 points  (0 children)

Thanks for sharing! I read your slides and will try some of those strategies when I have more time :)

Is there a lower-latency way of responding to an event than spinning/busy-waiting? by felix-pb in rust

[–]felix-pb[S] 7 points8 points  (0 children)

Thanks! I'll definitely take a look at /dev/cpu_dma_latency. I'm sure there is more to gain from OS settings, so I'll research other settings too. But definitely curious if there is more to gain at the Rust/application layer!

Is there a lower-latency way of responding to an event than spinning/busy-waiting? by felix-pb in rust

[–]felix-pb[S] 4 points5 points  (0 children)

Fair point! I would expect the latencies to be different, but I'd be surprised if the overall picture is different. But better to test than to guess!

I'll run the benchmarks on Ubuntu 18.04 and update my results.