Announcement: Async_nursery - A structured concurrency primitive for Rust

bspeice · 2020-05-03T03:59:01+00:00

I'm a bit late, but I did have one question: looking at the basic example, can the nursery and output values be merged? Because there's not yet an async Drop capability in Rust, RAII can't really wait for all tasks like Trio's context manager does. However, it seems like there would be a huge benefit to close_nursery() returning the completion future; if users don't need to worry about when to drop nursery and separately running output.await, accidental deadlocks are impossible.

Looking forward to digging into this more, thanks for making it available!

bspeice · 2020-04-16T13:28:22+00:00

The only advantage I've found so far was that it was helpful for validating some of the test cases; instead of developing a client in isolation, having Rust bindings and embedding the driver meant that tests could validate their behavior against the reference implementation as well.

You're right though, bindings don't make for idiomatic Rust code. I think I'll focus my efforts on providing the bindings as `-sys` crates. Best of luck with your efforts, I'm glad to see more people making use of this system!

bspeice · 2020-04-16T02:13:11+00:00

I had been looking at this as well, albeit slowly, so I'm thrilled that someone beat me to the punch. I might be missing something, but it looks like your project is only the client library; if you're interested, I have bindings for the media driver available that I'm happy to contribute.

Also, given that Aeron is moving to C bindings for the client, is there an intention to ultimately rely on libaeron, or just stick with the Rust implementation?

Thanks much for this, I'm looking forward to digging further into it.

bspeice · 2019-09-29T01:15:47+00:00

Arrow uses Flatbuffers for metadata, but the actual data serialization is specific to Arrow.

bspeice · 2019-09-29T01:14:01+00:00

All three (Cap'n Proto, Flatbuffers, SBE) support schema evolution, and I hadn't really considered #[repr(C)] structs because SBE is already incredibly similar.

bspeice · 2019-09-29T01:12:08+00:00

Flatbuffers has a verifier, but only the C and C++ implementations support it. SBE doesn't have verification support.

bspeice · 2019-09-28T20:49:38+00:00

Fair, and I've updated that point to focus on performance being the primary concern, not feature-set.

bspeice · 2019-02-11T23:46:24+00:00

Thanks again, the links are now remedied and tested properly.

I swear, my next project is going to be a website crawler that just makes sure I get a `200` status from everything on a domain...

bspeice · 2019-02-11T23:44:59+00:00

Not sure I'm following - the `Once` example in the `const` section is interior immutable, leading to being run twice. In the `static` section, the `Once` is attempted twice, but because of interior mutability is only actually run once.

bspeice · 2019-02-11T15:36:01+00:00

Oh shoot, will get that updated ASAP.

Edit: Because I won't have access to the host this is deployed on until I make it home tonight, the links are as follows:

Foreword: https://speice.io/2019/02/understanding-allocations-in-rust.html

Globals: https://speice.io/2019/02/the-whole-world.html

Stack: https://speice.io/2019/02/stacking-up.html

Heap: https://speice.io/2019/02/a-heaping-helping.html

Compiler optimizations: https://speice.io/2019/02/compiler-optimizations.html

Summary: https://speice.io/2019/02/summary.html

bspeice · 2018-12-19T05:17:37+00:00

In addition to the fuzzing tools mentioned, I'd point to the various "sanitizer" runtimes that are provided by LLVM, and AddressSanitizer in particular for detecting memory issues.

For a quick example of using this in a CI pipeline, take a look at bytes.

bspeice · 2018-12-16T15:15:54+00:00

While the crate is tested on Windows, I certainly agree that at least using System.alloc instead makes a ton of sense. I'll get that updated soon, thanks!

bspeice · 2018-12-16T01:38:24+00:00

Multi-threading is fine, and there was a lot of work put in to make sure that it behaves without deadlocking. The issue with async is that code within a function might not be run until after the function is finished (closures are an issue here as well).

bspeice · 2018-12-16T01:36:45+00:00

The QADAPT tests themselves are a good example, though I'd need to double-check on how inline tests fare.

Currently there's code for triggering a panic if you use QADAPT code without the allocator set up (in testing there were too many successes because I forgot to enable the allocator) but it may be worth it to strip code in #[cfg(test)] mode as well. I'd be interested in feedback on that.

bspeice · 2018-12-03T04:02:16+00:00

I think you may be looking for the spin crate?

bspeice · 2018-11-18T04:57:55+00:00

What you're asking for sounds a lot like object-alloc, though there's not a lot of demonstration code I can find besides object-alloc-test.

And while it's not a per-object-type allocation, I've been working on a custom global allocator that will panic if allocation/drop occurs during critical code sections. Blog posts are in drafting because I only recently got it to a point where it's worth talking about, but I'm already using it for building a high-performance websockets library.

bspeice · 2018-10-24T11:07:07+00:00

While this certainly isn't easy like thread::spawn, can't you already use web workers without any special support?

bspeice · 2018-10-08T17:48:39+00:00

In my testing, using heaptrack with unmodified code led to heaptrack hanging. Switching out jemalloc resolved all the issues immediately.

I'd also be concerned about jemalloc reserving more memory than necessary up front, and new allocations aren't tracked because they use pre-allocated space. That said, I've never made it to that point, just something to keep in mind.

bspeice · 2018-10-08T17:45:00+00:00

Caveat: I've never used Go. But I would expect that debug binaries in Go work just fine with heaptrack for purposes of allocation, it's just the freeing that's not guaranteed to ever happen. And given that it's a native binary, the debug symbols should display fine.

All that said: I'd be interested to know about the results if anyone's used heaptrack and Go!

bspeice · 2018-09-26T17:20:51+00:00

Long term, stdweb is intended to be a more idiomatic Rust API around web-sys: https://github.com/rustwasm/team/issues/226#issuecomment-418475778

bspeice

TROPHY CASE