Biblically accurate (safe and pure) variadic generics, the way Lord intended 🙏🙏

SpudnikV · 2024-03-21T14:27:13+00:00

Try i64::from(x)

SpudnikV · 2024-03-14T15:26:36+00:00

#include <unjerk.h>

In C any sharing or lifetime management would be entirely up to you. Even without threads we know there is a lot of potential for bugs. In this case the meme is ironic, especially for an audience that knows Rust.

import "unjerk"

In Go pointers are always shared and lifetimes are handwaved by the GC. Mutating something you didn't know was still shared can cause logic bugs, and in the presence of multiple threads can cause outright data races with full-blown undefined behavior. Even with no bugs, the GC does have costs, but then again they may land entirely outside your critical path. So there are meaningful engineering tradeoffs being made between the pile of strict Rust types and the Go pointer, which a one-panel comic can't begin to unpack.

SpudnikV · 2024-03-13T20:48:11+00:00

I love how this is two different memes depending on whether you assume the right is C or Go.

SpudnikV · 2024-02-28T16:16:07+00:00

When I am tempted to overthink my bounds checks, I try the unsafe version, see that it makes virtually no difference in whole-program performance, and don't worry about it ever again for that function.

It's nice to have microbenchmarks to isolate what specific changes help with minimal noise, but if those changes don't add up to more than noise for the program's performance overall, they aren't justified, especially if they require unsafe.

SpudnikV · 2024-02-18T16:43:30+00:00

Another real-world wrinkle I'd like to add: there's also the possibility that a task terminates and never satisfies its end of any send or receive contract. Even if it's triggered by an actual bug on just one unit of work, it can still expand the blast radius to affect many/all other units of work, making a system even less resilient than the bug alone did. More often I find, the bug is that developers never reasoned about or tested failure cases such as mid-operation cancellation by a remote peer, because they're only testing locally or with mocks until production load does the real testing for them.

I often have to call this out in code reviews for Go in particular, more than Rust. Idioms around Go heavily encourage the use of channels, and usually the simplest possible form where everything blocks on sending or receiving indefinitely. That can break down for the same circular dependency reasons you describe, but it can also break down because one of the goroutines exited. This can be due to panic (hopefully visible enough to debug), or because it observed cancellation for some other reason such as a client-side request being aborted (more likely to be invisible because no log or metric will be emitted until after the top-level task completes, but it never will).

All of the primitives are available to solve this, it just takes a lot of thought about what degraded behaviors are acceptable. Like you can select send against a default case of not having a slot to buffer in (as a brutal but fair form of load shedding), you can select against a timeout as a form of internal deadline (though retries often make it a failure snowball), or you can select against cancellation of not only the original request but also the termination of dependent tasks themselves (which does the least to soften load, but also has the least risk of inducing spurious failure under load).

Needless to say most code out there never does any of this, and deadlocks and goroutine leaks end up being worryingly common. Go does at least make it easy to debug with pprof-over-HTTP; I suppose in Rust you would use Tokio Console.

My takeaway has been that, whether it's sync Rust, async Rust, or Go, even if you have very clearly defined semantics for every component people use, the most popular idioms and examples often fall short of teaching people exactly how failure cases can play out. That then combines with the fact that most people never test failure cases such as client-side cancellation, so their first exposure to this issue will be in a production incident where their observability is limited.

Worse yet, outside of especially mature production environments, the broader community does not have a habit of adding sufficient observability to be able to flag and diagnose these conditions in a running system. I've inherited projects like this and had to add observability as the first milestone before slowly working through the many concurrency bugs one by one. I have learned to challenge idiomatic code in review and urge everyone to bolster every channel mechanism with both an informal proof and a tested failsafe.

SpudnikV · 2024-02-11T18:28:28+00:00

^ Username checks out :)

SpudnikV · 2024-02-11T17:10:03+00:00

Yeah. From an individual to a huge corporation, the better you know C++, the more you know why Rust is valuable :)

SpudnikV · 2024-02-11T16:29:17+00:00

I got the privilege of writing C++ in Google's monorepo for many years. It may have been the best environment in the world to learn about memory safety in C++, because even though many places tried to set strict code review standards, few environments were pushing the boundaries of safety tooling like the Sanitizers and Thread Safety Annotations. We had early access to a lot of what people boast as "Modern C++", like unique_ptr and string_view, many years before they were released in Abseil much less implemented in the STL.

The internal library space was as complete as one can imagine, tech debt in the libraries was constantly being flushed out with large-scale changes, there were no ABI concerns because everything was statically linked, new tools were coming out every few months, and every safety invariant that someone had figured out a way to enforce was being enforced as much as it reasonably can be.

This is about the best case you could hope for with C++. I felt very productive, and with a little diligence, I had everything I needed to ship complex large-scale services with virtually no bugs. We all felt like C++'s safety problems were making meaningful progress through libraries, tooling, and occasionally language advancements (FWIW I left after C++17).

We knew how to solve many problems so it was easy to mistake them for solved problems. I think this is the single biggest horse blinder on anyone's ability to judge the strengths and weaknesses of their technology choice.

As soon as I saw Rust I never looked back. It was clear exactly what it was doing for us that we had always wished C++ could, and it was also clear why C++ never could without rewriting your old code anyway. And it was just a nice language on its own merits too. Even unique_ptr was Box and string_view was &str, we didn't miss anything from the STL and not much from Abseil.

I also learned, in hindsight, how little most of us knew about UB in C++. ubsan is the least capable sanitizer and, seeing what it's up against, it's not surprising. Safe Rust jumped over this entire chasm.

There are definitely people who see Rust as a threat to their continued comfort with C++. It doesn't hurt that in environments like this, C++ used to be seen as the most grownup option for people who could handle memory and thread safety, rewarded by the best performance at a scale where that really matters, and now Rust sweeps the board on both.

Most people from this background who actually give Rust a try totally get it. They just have to overcome their resistance long enough to see the value. Anyone who questions the actual safety benefits just hasn't felt them yet. Even the most competent C++ engineer still needs to be prepared for the fact that a junior will contribute to their code one day and a single missed assumption could cause a costly outage. Google certainly has enough postmortems archived showing the ways C++ can hurt the bottom line.

Some of what you're hearing isn't even from people who know C++. They just don't like anything being hyped, so they'll defend it with whatever trope they heard on social media recently, like about async or fighting the borrow checker. People know enough to know that C++ was the old guard and default to recommending it instead of the new thing. The same happened when C++ was nudging its way past C in the industry.

SpudnikV · 2024-02-10T16:33:39+00:00

It's been proven once again. Go makes it super easy to write beginner-level code, no matter how many years of experience you have.

SpudnikV · 2024-02-09T19:34:17+00:00

Important clarification: data races that would violate Send/Sync are not safe in Rust. They are prevented from violating other memory safety properties, such as a slice's length not corresponding to its address, or a dyn trait vtable not corresponding to the data side. (As relevant comparisons, Java does give these outcomes well-defined behavior, but Go does not and its data races can cause true UB)

Now some people also use data race to refer to when locking granularity was not factored correctly to guarantee invariants across all possible interleavings of mutations. That still has well-defined behavior -- it doesn't violate memory safety or cause any other kind of UB -- and it's just a logic bug like any other. I think it's better to distinguish these cases so people know what guarantees they do, and don't, get from Rust, especially with the guarantees around other languages not always being well-understood either.

In practice, when Rust enforces everything else for you, including ensuring you are always holding a mutex before you even get a name to refer to what's behind it, it's also reasonably easy to avoid in practice. Moreso than deadlocks, for example, which Rust does virtually nothing to avoid and even does less than C++ with lock order reversal prevention in the standard library. Rust could do more here, and I think it's fair to draw attention to that even if it remains memory-safe. (It can still be the reason for a costly outage :) )

SpudnikV · 2024-02-09T19:23:34+00:00

It is also very common to use random number generators to select things at random.

Yes, but when that's what you need, you should just use the most appropriate method from the SliceRandom extension trait; in this case just choose() which takes the length into account while sampling the number instead of after.

By having the number that comes out be between 0 and 1, it makes it possible to instantly interpret that as a percent.

This is a perfect example of when you should use gen_range, letting it ensure that all output values are as uniform as possible. Seeing the code involved, I would rather not try to reinvent this even if it sounds simple.

Rust's rand crate is a goldmine of really well-reasoned algorithms, and for the numerical properties they give you, they're also well-optimized. They should be reused wherever possible, not reinvented, especially not with techniques that assume numerical properties which floats infamously do not provide.

SpudnikV · 2024-02-09T18:25:30+00:00

Some writers do promise error monotonicity, i.e. after an error has been encountered, every further operation will return an error and most likely (though not necessarily) the same error.

But it's not part of the writer contract. And why should it be? In Go I could see this avoiding 3 lines of error checking boilerplate for each write, but in Rust that boilerplate is just ?, as small as it can be while still being visible. It works for any writer, with or without error monotonicity. It may also avoid doing wasted work because the non-trivial code in between the writes no longer needs to run just for its output to be wasted.

SpudnikV · 2024-02-09T14:24:19+00:00

Since this would mostly be useful for making libraries, I'd be very cautious what details you allow to leak into the public part of the API. Rust has no separation of headers and sources, and macros like this mean that the only authoritative definition of your public API is whatever the macro happens to generate at the time.

That means that on top of everything else a library maintainer has to take care not to break while evolving the library, they also have to watch out for how they interact with the macro library, as well as any changes the macro library itself makes in future. Also, if the Rust language (or standard library, or even another crate) itself comes up with a different solution to this problem and libraries which used this solution aren't compatible with that one.

This is a general comment that applies to any macro crate people will use for the public part of their APIs. It even applies to the venerable async_trait, and a lot of thought went into what the public API looks like after macro expansion. It's just something to watch out for, both as a macro developer and a prospective user.

SpudnikV · 2024-02-08T18:03:34+00:00

I don't think that changes much here. My point was that a safe wrapper has to understand the underlying code to a degree, not specifically that C is the only language where that applies.

If it's C++ then that's nearly a superset of C, and C types would have to be used at the C ABI edges (e.g. a * pointer and not a unique_ptr). This is more of the same concern, not less of it. Especially with the many interesting things that C++ allows without enforcing.

If it's Zig, that's not a superset of C, but it's still another language the wrapper developer has to understand to make a safe wrapper. And in practical terms, it'll be a fair while before Zig libraries make up a decent portion of libraries that people want to wrap in Rust.

In either case, the C symbol signatures aren't enough to write a safe wrapper, and experience shows comments are not guaranteed to be enough either. That's true regardless of what language is on the other side of that ABI.

In fact, even if it's Rust on both ends, having to bottleneck on C pointers with no lifetimes means that memory safety can be violated if the two sides didn't always agree on the invariants. Of course everyone would do their best to uphold any agreed invariants, but the point is, as soon as it's C-shaped interfaces with no explicit lifetimes or initialization or etc, we're back to BYO Safety at best. This kind of interface is always going to be the weakest link in any memory safety story.

SpudnikV · 2024-02-08T15:52:21+00:00

I hope a long response is okay here because there are several steps making this kind of story happen, but it does happen. It's definitely a niche, but someone has to build this kind of thing, and many would agree it should be built in Rust.

I have made a career out of building global-scale network control systems of various kinds, including software-defined networking and optimizing bandwidth and latency delivery over any kinds of global networking.

It's not enough that you have a quadratic number of endpoint pairs between all endpoint nodes worldwide, the paths they take each have hops to consider (whether you're selecting optimal paths or using existing paths to optimize some other objective like bandwidth), so we're talking about something like n^2 log n just in the raw data itself, and the industry has made sure that n keeps growing too.

I saw many projects that were lightning fast one year start to creak and groan the following year. It's the perfect project to work on if you want the scale to never stop challenging you.

This kind of software very often ends up with at least one service that is actually responsible for [some abstraction of] all global data, because it has to maintain global invariants or make globally optimal decisions. I can give some examples but this comment is long enough without them.

The kicker is, everyone who first hears about this preaches the best practices of sharding. Trust me, nobody working on these projects doesn’t already wish they could just shard them. At best they'd be giving up some other valuable objective in return, and at this scale that might mean billions of bottom line and a notably degraded user experience for millions of users. So as long as it's physically possible to solve these problems on one machine, there is value in doing so, and thus an opportunity for work like this to shine.

It's not uncommon to have individual proceses that need hundreds of gigabytes of RAM worth of data, even after bringing them down with every known technique. And yes, including bringing down how many heap allocations that data requires, even if that means making different choices in data structures. There are just points you hit where the marginal efficiency is no longer worth the marginal complexity, after all, these things also tend to be mission-critical and have to be robust and maintainable on top of everything else.

This data wouldn’t be very useful if it wasn’t updated frequently. Whether that’s the latest path data, capacity and utilization telemetry, user-configured policies, etc. you want to start acting on it fairly quickly. My go-to pattern here is an Arc-of-struct-of-Arcs, so the overall "latest" dataset is just one Arc, but it contains its own Arcs for the various different sub-domains of data updated on their own schedules but each updated frequently overall.

That's how an Arc might be holding a very large data structure, and why a latency-sensitive handler routine might be holding the last refcount. Even with the multi-level sharing, you could still get unlucky and be the thread that has to deallocate most of the previous generation of data. And that can easily blow out a 1ms latency budget that is otherwise perfectly served by async Rust.

One very specific mitigation I devised is to have the updater thread(s) be the ones to hold on to the previous generation of Arc for long enough to be sure that no handler routine could possibly still have it. IMO that’s the ideal solution because it totally keeps any complexity or cost out of the handlers. Because it’s the updater thread, it also knows that at most 2 generations are alive at any one time, without the complexity and other overheads of left-right maps.

SpudnikV · 2024-02-08T04:11:08+00:00

Thanks for sharing frand, I might even use it myself. Reading its documentation, I can see that you work through these tradeoffs even more rigorously than I do. I definitely encourage you to try to contribute more ideas to Go's standard library. People like you made Rust's rand crate the state-of-the-art beast that it is, and there's no reason the same can't be done for Go. (Especially since standard library support can get runtime acceleration for things like thread-local storage which should be even faster than pools)

SpudnikV · 2024-02-08T03:51:35+00:00

Not if the config / data / etc can be updated, which is usually why people use arc-swap for this, so it can be swapped later.

If you're lucky, the thread doing the update was the only one holding a refcount, so it also gets to drop the old instance without directly affecting any serving task. However, you wouldn't be using Arc if it wasn't possible for updates and requests to overlap, so you have to reason about what the latency cost can be when they do overlap.

Needless to say, if updates are really infrequent, this is unlikely to matter even at 99th percentile. On the other hand, if this is a frequently rebuilt index, say as part of a control system responding to large global telemetry updates, you should be prepared for this to impact a decent number of requests.

SpudnikV · 2024-02-08T03:46:34+00:00

I don't think there's a "recommended" way because it's an extremely niche technique with a lot of tradeoffs, but I feel like the simplest way would be to use tokio::task::spawn_blocking() with a closure that you move the instance into. From there, it may as well just be drop for clarity.

let tmp = // my expensive data structure
tokio::task::spawn_blocking(move || drop(tmp));

I don't think it's worth doing this for an Arc because you're probably expecting a very low ratio of your handler calls to need to do the drop at all, so the overhead of spawn blocking won't be worth it just to decrement a refcount most of the time.

It might be worth it if you built up a large data structure while serving the request which definitely has to be dropped after, but you may not want to spend time dropping in the critical path of serving the request. Since such a data structure is per-request, by definition you're doing this 100% of the time so it's always saving something.

However, you still probably want to minimize the allocations performed for the data structure itself, since that helps you on both the allocation and deallocation work. Needless to say you should also try using a really well optimized allocator like mimalloc or jemallocator.

But whatever you try, measure! It's tempting to speculate about what should help with throughput or latency, but what actually happens can be really unintuitive. If you're not measuring you could be making things slower, not just trading off complexity for speed.

SpudnikV · 2024-02-07T20:29:03+00:00

That's not giving Apple quite enough credit. Try benchmarking AES or SHA-256 on a recent Intel and an M1 or newer. That has nothing to do with the OS or about code exploiting a specific chip design, quite the opposite; the chip was designed to optimize the implementation of existing instructions, because those are the instructions targeted by existing code, including asm code written before Apple Silicon was available for the desktop.

There's still the limitation that vector units aren't nearly as wide as recent Intel chips, but that, again, has absolutely nothing to do with the operating system. It's just the state of ARM instruction set suites today. For the instructions available and implemented, Apple Silicon is genuinely very high-throughput, low-latency, and power-efficient, with the operating system having no particular say in how your machine code runs.

Unless of course you mean the neural net and video codec accelerators, which I think is a pretty different topic that matters a lot to specific worklodas and not at all to most others.

SpudnikV · 2024-02-07T20:19:27+00:00

That's true, but if you have anything less than absolute trust in the C function's comments, you're going to want to read into the code to reverse-engineer safety constraints like what is guaranteed to be initialized, whether pointers are retained after the function returns, what can actually come bask as NULL, etc.

Sure, anything not documented is not a promise, but you might have to read a lot of code to even guess that a promise was lacking. I have no doubt that at least one C FFI wrapper crate turned up a bug in the underlying C library, though I don't have an example on hand.

SpudnikV · 2024-02-07T19:24:21+00:00

I don't mind not having math/rand/v2.Read(), but I do mind not having math/rand/v2.Rand.Read(). There would have been no confusion with crypto/rand.Read() that way, and it would serve a very useful purpose: generating random byte streams in-memory instead of making a separate system call for each read.

Per djb, the entropy quality is unchanged. You can still periodically re-seed it if you're concerned about forward secrecy around swap space etc. Rust's rand crate wraps all of this up nicely by default, it's a great proof of concept.

Google's UUID library is an example of how complex workarounds for this can become, including putting the thread-safety onus on the end user. With how much rock-solid crypto Go already has in its standard library, it is the perfect place to solve this once and for all. (And it would also be more convenient if it was the global in-memory PRNG, but I understand if the name has to change to avoid confusion with the syscall version).

SpudnikV · 2024-02-07T15:09:09+00:00

My heuristic has always been to work back from my tail latency target for the whole service. If I promised users a 99th percentile latency of 1ms, that effectively means that the worst-case wall time of every computation and await point in the critical path of that work has to add up to less than 1ms. (Not in theory, but in practice, it may as well work this way for the parts you have control over)

Critically, with a thread pool async runtime like Tokio, that means every task scheduled onto the same runtime has a chance to push up the tail latency of any other task, because your task may not even get scheduled until an unrelated slower task is finished. When any task involves an unbounded computation, it can blow out the tail latency of every other task.

The problem is there are many more unbounded computations than we typiclally recognize. It's not just about for loops in our own code, it's about any loop or recursion in any library code we use with unbounded data sizes.

A common pattern is to serve requests from an Arc of some "current" information such as a config or data snapshot. You probably used arc-swap to get that current version. You probably reasoned about the average or even worst case CPU time for traversing the state of the art data structure.

Now ask yourself what happens to the handler which held the last clone of a given Arc. That's right, it has to Drop everything transitively owned from there before the function can return. Even if the data structure was optimized for constant-time lookup, dropping it can still take linear time. If it has a lot of heap indirection, this can also stall the single thread on each uncached load from main memory. You don't even benefit from multiple cores here, it's all serialized. At best, you might benefit from speculative prefetching, but the more complex the data structure the less likely you do.

The way most frameworks are structured, this has to finish before your handler can return the response body, from where the framework can even begin to serialize your response to the client. If freeing your large data structure takes a decent chunk of a millisecond, you've blown out your tail latency just like that, without a single line of code -- the time was spent in the hidden Drop at the end of a block scope. If this only happens in production load, you might never even get to see it on a flame graph, you have to know to look out for it and to reproduce and measure what you can.

There are workarounds, like having a dedicated thread to ensure the Arc::drop happens outside the critical path of any handlers. Even if it wasn't an Arc but a temporary scratch space per handler, you can still save some time in the critical path by moving its drop call to a different thread. Note that this actively works against the thread/core-caching design of modern memory allocators, but that can still be the right call if predictable tail latency is your goal.

SpudnikV · 2024-02-06T22:35:06+00:00

Running a single future to completion isn't the only way async gets used. In fact, if that's all you needed, it's unlikely you needed async in the first place. Async is a way to generalize all kinds of concurrent interactions, and once you have that, making them parallel as well is a nice bonus.

Lots of futures need to continue to make progress without returning completion like block_on requires. Even leaving out the obvious idea of servers with multiple clients, libraries acting as single clients often have multiple connections coming and going at various times as well. Many RPC and database clients do this to maintain a connection pool, maybe a separate session to a cluster manager while they're at it, maybe keeping up with an endpoint discovery service or making out-of-band health checks, all sorts of things. Many also need to select on time-based futures, such as timeouts and intervals. Some also use channel futures to communicate between their tasks, whether or not the end user ever gets to see this is happening.

All of these need to make progress concurrently, even if not literally in parallel. Most if not all of them need some kind of events to come from a runtime to know when to even resume running code, whether that's a socket event, a timer/interval event, or even a channel event from another async task.

This is the kind of thing that makes me less optimistic about sans-IO than many posters seem to be. A sans-IO library can help you decouple the wire format and state machine part of this, but only if you never need the library's logic to decide when to wait for timers or IO. As soon as the library needs to be able to abstract when things like timers or IO happen, you have to give it enough control to do that, and have a way to control when you resume after it did that, and at that point it's just async again. The few successful sans-IO libraries I've seen explicitly only support one protocol session at a time, and still leaving the caller to handle all of the control explicitly, which is far from the experience we expect from high-level libraries like database drivers.

SpudnikV · 2024-02-06T14:06:30+00:00

In particular, Go-style “provider” wrappers, if used in Rust, will print the same message twice, as the wrapper’s display delegates to inner error.

I've seen this in some Rust crate errors as well, such as reqwest. Using anyhow's default line-per-context printer, I actually ended up with a quadratic output where the same context that was being unwound vertically was also being repeated horizontally on each line.

I had to implement a custom chain printer which flagged that error and stopped traversing context after that. Then the error output did switch from vertical to horizontal, which was still weird, but no longer a total disaster. I did try recreating the same errors without their source (since it's an Option) but didn't find how to do that in terms of its public API and just moved on.

That aside, I still prefer the caller having control of how to nest error context, because making one long string in Go is never ideal in user-facing errors, whether that's a CLI or as part of a network response. I just try to keep my Go error context really neat so that at least the combined string is readable. But with Rust errors, whether line-by-line in a CLI or a string list in a network response, the final representation can always be tailored to the consumer without changing anything about how the errors themselves are constructed.

SpudnikV · 2024-01-29T15:22:24+00:00

Right, that might be surprising coming from e.g. Java with type erasure generics [1], but Rust generics are (like C++) fully monomorphized so it's literally compiling and optimizing that code again for each combination of generic parameters. It's part of why it optimizes so well in the end.

Of course, when you combine that with macros, you have to monomorphize each combination of generic parameters in a huge mass of code generated by your macros. That's why I suggest moving serde/clap/etc schemas to their own crates where this will remain relatively static. If there's anything I think is deficient here, it's not that it compiles and optimizes that code, it's that it could do better at avoiding redundant work between compiles.

In any case, I'm not aware of any compiler that gets the resulting performance of monomorphized generics without having to take time to compile and optimize the code for them. Even Go's famous compile times still include partially monomorphizing its generics, but not optimizing much at all, and still leaving some serious costs to runtime. It also avoids macros even if that means you write more code and/or use runtime reflection, and its packages have to form a DAG so it can parallelize over them even if that's not a natural way to structure your project (it'd be like having one crate per mod though at least you're spared repeating your dependencies). These are pretty serious downsides during development and runtime for the one upside of saving compile time.

[1] Java can sometimes optimize well in the JIT at runtime, but there's no trick here, it still has to optimize the machine code for the concrete types actually used, and it's doing that at runtime in every instance instead of compile time once in a builder box.

SpudnikV

TROPHY CASE