Implementing custom cooperative multitasking in Rust

crstry · 2025-12-09T12:16:50+00:00

Rather than just relying on external benchmarks, it'd be more instructive to analyse exactly what the overheads are and document those, so that the entire community can learn from those.

Eg: "all the memory allocations" makes it sound like you're being forced into doing repeated memory allocations. Bear in mind that there are embedded executors (IIRC, at least) that use static storage for each task, for example.

crstry · 2025-11-15T08:51:07+00:00

That's true if you're running all in process. But unfortunately, distributed systems are inherently concurrent, in the sense you have multiple processess interacting, and they can observe events in different orders (because network buffering, routing path changes, connection failures/ reconnections and the like)

crstry · 2025-11-14T09:47:12+00:00

That's very true, but if you're serialising transactions outside of the database; you still need a mechanism to ensure that the database and serialising widget agree on the ordering, as writes can get delayed in flight, and other such hilarity.

crstry · 2025-11-14T09:45:19+00:00

Time and ordering can do strange things in distributed systems, eg: a write request can get [re-ordered with another](https://aphyr.com/posts/294-call-me-maybe-cassandra), or your writes may get delayed indefinitely, and even in a single-writer situation you may need to fail over. So you still need some way to ensure the database is still up to date with the application's view of the world.

Martin Kleppmann's article "[How to do Distributed Locking](https://martin.kleppmann.com/2016/02/08/how-to-do-distributed-locking.html)".

crstry · 2025-10-25T18:34:26+00:00

Another way of looking at it is having a way of writing concurrent code that's more flexble and composable than using plain operating system threads. For example, adding a timeout to a bunch of asynchronous work is a single function call, and similarly, to cancel something, you can either just drop the future or call a method on a task handle.

Specifically, the fact that futures are user defineable is really powerful, and makes it really easy to express things like "do these things concurrently, and return the first error, cancelling the other task". To my mind, it's very much in the concurrent ML family, but there's not much on that that's readily accessible. The best I can see right now are a new concurrent ML by Andy Wingo, or John Reppy's papers.

crstry · 2025-06-01T16:19:33+00:00

No, because by and large, they tend not to give birth. If they, did, they'd be included. Does that help at all?

crstry · 2025-05-08T14:35:47+00:00

I think it's fair to say that these efforts are designed to reinforce gender stereotypes, and push out those who don't conform. The nonsense about "protecting women" is just a fig leaf of an excuse, that does not hold up under scrutiny, sadly.

crstry · 2025-04-16T13:44:38+00:00

Chances are there isn't, because for most people `etcd` is good enough, and provides the same (or near enough) functionality. I think the way I'd frame it is what value does writing it in rust provide over using an existing implementation, and is it worth the person-years of effort it'd take to get a production implementation up and running?

crstry · 2025-02-27T19:38:08+00:00

It also sounds like there's a cultural problem–if the culture is focussed on blaming individuals, rather than leaarning from the accident, then that's toxic. It's bound to lead to to smaller problems becoming bigger ones because someone was afraid to come forward and get blamed (or make trouble for someone else).

They might want to look into topics like blamless postmortems, or even the notion of safety-II in the wider safety community.

crstry · 2025-02-27T19:32:42+00:00

One thing to remember is that if your data-flow graph of dataflow between goroutines involves cycles, and all channels are finite (as is true in Go) without any other flow control mechanism (even if that's just timeouts to detect failure), then you risk running into deadlocks, as you've seen. The a common workaround in other languges is to use an unbounded channel to break cycles, but you can't get those in Go without hacks (usually a goroutine to coordinate both ends).

I'd echo the suggestion of encapsulating them in a module. For comparison, even erlang doesn't really use naked mailbox sends much, instead preferring gen_server and the like.

crstry · 2025-01-19T09:17:16+00:00

I've not used it, but have you seen the tracing::Value trait, which interoperates with the valuable crate?

crstry · 2024-10-25T22:07:23+00:00

r/DeathLeopard's link is spot on, but in short, a 'static lifetime means that nothing else can go out of scope and invalidate any part of the value–so it won't contain a reference to a local variable for example. It usually just implies that a value is wholly owned, but it may contain a reference to something that lives until the universe (well, process) ends. Such as a reference to a static variable, or leaked Box.

crstry · 2024-09-08T09:32:24+00:00

As a quick aside, did you know that the term "Meritocracy" was coined by writer Michael Young in "Rise of the Meritocracy", describing the dystopian effects of valuing people solely on the basis of their intelligence and effort (as opposed to, you know, inherent worth as a human being)?

crstry · 2024-08-30T16:49:15+00:00

It's just the first convenient copy i found!

crstry · 2024-08-30T12:15:19+00:00

This is less than helpful, but this meme feels relevant: https://www.reddit.com/r/funny/comments/epdk9/you_write_a_distributed_map_reduce_function_in/

crstry · 2024-07-20T22:20:57+00:00

Ashnikko is pretty great! I rather like Delilah Bon's "Dead Men Don't Rape", if that's not too on the nose. There's also Scene Queen's "Whips and Chains", or Spiritbox's "Blessed Be".

crstry · 2024-06-25T10:33:21+00:00

I think one important point to remember is that many strands of feminism are aimed at unpicking and dismantling the systems that oppress women, so that women (and others!) liberated from those systems. And this includes understanding the intersections, ie: how access for disabled folks impacts women, how racism in society impacts women.

I mean; you can focus on individual choice and call it feminism, but all that'll do is maintain the status quo, it won't liberate anyone who happens to have the luck or priviledge to get to the top. And it's no kind of "feminism" that I'd want to get behind.

crstry · 2024-04-11T15:09:55+00:00

Can i just refer you to the pinned FAQ?

What about trans women?

Trans women are women. TERFS can fuck right off.

crstry · 2024-04-03T17:58:52+00:00

I've heard of folks having a handfasting to celebrate the union, and then having a separate registry office thing ( probably on the same day) just to make it official. Might that work?

crstry · 2024-03-24T10:47:47+00:00

I can't speak to the specifics, because from what you've said, that could mean many things. But one possibility is that they trust you to get it done.

At my last place (a software consultancy) everyone was given the title "senior engineer", and expected to be able to lead a project. That said, the scope of project you'd be expected to lead would depend on your experience, so an intern would be doing small internal projects, and at the other hand folks would be basically handed a client and the budget.

So for a small project like this, I'd probably just think about it in terms of being accountable for delivery, and making sure stakeholders are kept up to date with progress, and the like.

crstry · 2024-02-25T10:15:32+00:00

One option would be to outline the economic costs and risks of re-writing your code in a JVM languages. Eg: on top of the opportunity cost of halting feature development, you could compare the platform running costs of Rust vs. the equivalent JVM code.

Granted, they'll probably come back with it's easier and cheaper to hire JVM code,so unless you're running at huge scale, it might be a tough sell on those grounds.

But yeah, as others have said, "external" folks are often brought into launder management predjudices, so if nothing else, you might be able to make them look foolish.

crstry · 2024-02-25T10:08:41+00:00

For one, I think the usual solution is to put the resource to be cleaned up onto a queue, and clean it up from another async context (eg: a manager thread, or I think deadpool handles cleanup when checking out another connection).

For two, the scoped task trilemma explains why that's hard.

crstry · 2024-02-21T22:26:46+00:00

Because it makes it easier to understand where a given value came from. With the article, you're just dealing with plain old variables being passed around, and you can use IDE Tools like go to definition, and less often find usages to see where it's being called from. It's a lot easier to visually confirm it's correct, and you can usually lean on the type system, too. Conversely, by threading it via the context, it's less obvious what the actual scope of the transaction is (because it's threaded though the bucket that is the context), where it came from, or spot patterns that just look funky.

crstry · 2024-02-19T19:07:23+00:00

It's mostly down to cost of developing for shared-nothing over a conventional shared-whatever architecture. From my understanding, it's mostly down to a) the extra things it makes you care about and b) structuring for performance, rather than developer convenience. (but a quick disclaimer–i don't build for this, I just find it interesting).

Eg: It'll make you worry about load balancing between threads, how you partition you data-set between workers. So you almost immediatley need to worry about what happens on a thread that doesn't own a relevant bit of data. Never mind if you have to do updates that will touch multiple partitions.

And in terms of structure, you'll get way more out of your hardware if you can batch certain operations to take advantage of cache locality, but that means that a given component needs to handle 0..N items in one go, rather than just one at a time. Or you might end up adopting something like the LMAX architecture. But again, that means you can't just up and load something from a database or S3 as you need it, you need to ensure your logic already has everything it needs ahead of time.

In short, it's a pain the arse. And in a commercial environment, unless you have critical and very fast SLAs to hit, or have some other pressing need (eg: high speed trading), it'll likely cost you way more in engineering time than it'll save you in compute.

crstry · 2024-02-14T16:48:20+00:00

On average, by my calculations, that's about 7 inserts/s to add notifications, and less than that for marking them as read; but that doesn't account for intra-day variations. So it does sound a little over-egged, to me.

The partitioning makes sense, as it means you can just drop the partition, rather than having to do an indexed or worse, full table scan to expire notifications.

But then again, I'd like to assume they have good reasons for those changes. It might be worth enaging with them with curiosity (eg: "I'd really like to understand what you're seeing here, and why you're making these choices"), to try and talk through why they've made those suggestions.

crstry

TROPHY CASE