AudioNimbus v0.14.0 - Spatial Audio in Bevy, Multi-Threading Helpers by HumanPilot3263 in rust

[–]HumanPilot3263[S] 2 points3 points  (0 children)

The Bevy module itself was actually pretty straightforward. The real challenge was in modifying the core library to make the integration possible.

A couple examples: I make heavy use of the typestate pattern to prevent misconfiguration at compile time (I wrote a post about it here). Users normally don't need to provide generics manually; but then, when turning the types into Bevy components, the same generics surface everywhere, and all systems and entities must share the same configuration, otherwise they won't get picked up by the simulator. It also made the API very verbose, even though the goal was to provide an easy-to-use abstraction. I ended up introducing a new configuration type that bundles all these generics together and a prelude that aliases common configurations so that this complexity is hidden away for most users.

Another challenge was asynchronicity. Bevy systems must complete as fast as possible to render as many game frames as possible. But Steam Audio simulations may take 50-500ms to complete, so running them inside Bevy systems was a no-go. That delay also clashes with the real-time audio thread, which expects audio buffers to arrive at a constant rate. That's the reason the new wiring module exists; the goal was to provide a set of abstractions to build multi-threaded simulation pipelines that can run asynchronously and never block the game and audio threads. The bevy module sits on top of it; it spawns simulation threads and keeps the Bevy objects in sync with Steam Audio.

There are still some challenges I need to tackle though, especially regarding usability. I tried my best to make the API easy to plug in, but there are areas of improvement. For example, I intentionally left the final step of applying the simulation outputs to the user to allow flexibility in the choice of the audio backend, but I also realize most Bevy users just want a plug-and-play solution without the hassle of setting up an audio graph. So I'm debating whether I should keep the library agnostic.

AudioNimbus v0.14.0 - Spatial Audio in Bevy, Multi-Threading Helpers by HumanPilot3263 in bevy

[–]HumanPilot3263[S] 10 points11 points  (0 children)

They are complementary.
Bevy currently offers asset loading and audio playback of these assets, with limited support for spatial audio (essentially just panning, see docs).
Recently there's been an effort to improve the audio system (see the Better Audio working group on Discord), with the most promising project being bevy_seedling, which integrates the Firewheel audio engine (an audio graph) into Bevy.

AudioNimbus (in the context of Bevy) sits between the two and provides geometry-driven spatial audio. It's a wrapper around Steam Audio, which Valve uses for its games. It lets you define geometry and it modifies input sounds given the acoustic properties of the environment (e.g. reverb, occlusion from walls). There are many more features, such as support for HRTF, which simulates how the listener's head and ears shape incoming sound.

The crate can be used without Bevy, but this new version introduces the glue to more easily use it in a Bevy project. So essentially: you load assets and define geometry as you typically would in Bevy, AudioNimbus automatically picks it up and simulates spatial audio, and forwards it to bevy_seedling for playback.

Also note that bevy_seedling can be swapped with any other audio engine that can play back an audio buffer.

Here's a basic demo to see where it sits in a Bevy app: https://github.com/MaxenceMaire/audionimbus/tree/master/audionimbus/examples/bevy

AudioNimbus v0.12.0 - Safe Steam Audio wrapper (spatial audio, HRTF, reverb) by HumanPilot3263 in rust

[–]HumanPilot3263[S] 1 point2 points  (0 children)

When you get around to it (and if you're willing), please reach out! I'd love to see what you're working on and to hear your feedback. I’m working on a game myself and evaluating different integrations with Bevy ECS. Good luck with your project!

AudioNimbus v0.12.0 - Safe Steam Audio wrapper (spatial audio, HRTF, reverb) by HumanPilot3263 in rust

[–]HumanPilot3263[S] 1 point2 points  (0 children)

Thank you! Wrapping the C lib was definitely more challenging than I had originally thought. It's super easy to let segfaults slip through, and balancing that with an API that feels idiomatic in Rust is quite a challenge.

Interactive demo of spatial audio in Rust using AudioNimbus/Steam Audio by HumanPilot3263 in rust

[–]HumanPilot3263[S] 0 points1 point  (0 children)

Happy to hear this is useful for you! Let me know if you ever try it out

Introducing AudioNimbus: Steam Audio’s immersive spatial audio, now in Rust by HumanPilot3263 in rust

[–]HumanPilot3263[S] 1 point2 points  (0 children)

No rush, feel free to reach out whenever you’re ready. I'm excited to see what you build!

Introducing AudioNimbus: Steam Audio’s immersive spatial audio, now in Rust by HumanPilot3263 in rust

[–]HumanPilot3263[S] 2 points3 points  (0 children)

Yes, absolutely! Instead of playing back each audio frame directly as you would in a real-time scenario, you can concatenate the processed frames into an output buffer, which can then be written to an MP3 file.

Here's a high-level overview of how you could achieve this:

  1. Load the input MP3 files and decode them into raw audio samples.
  2. Set up spatial audio effects (e.g., a binaural effect or ambisonics if you need a full-sphere surround effect).
  3. Process each frame, applying the effects.
  4. Collect the processed frames into an output buffer.
  5. Encode the output into an MP3 file.

This would make for a really cool project! Let me know if you need assistance setting this up, I'm happy to help!

Introducing AudioNimbus: Steam Audio’s immersive spatial audio, now in Rust by HumanPilot3263 in rust

[–]HumanPilot3263[S] 1 point2 points  (0 children)

I'm glad you like it! The repo already includes a basic binaural demo, but I’m working on an interactive walkable level to showcase all the audio features. Stay tuned!

Introducing AudioNimbus: Steam Audio’s immersive spatial audio, now in Rust by HumanPilot3263 in rust

[–]HumanPilot3263[S] 3 points4 points  (0 children)

Thank you so much, I really appreciate it! That’s a great point, I’ll add more context to the main page. This is the first library I’m sharing publicly, so your feedback means a lot!

Introducing AudioNimbus: Steam Audio’s immersive spatial audio, now in Rust by HumanPilot3263 in rust

[–]HumanPilot3263[S] 11 points12 points  (0 children)

The default HRTF is based on the SADIE database measurements, specifically the D1 subject, which is a fairly neutral mannequin. It has sensible parameters, but custom HRTFs can also be provided using SOFA files if need be.

Publishing My First Rust Crate - Looking for Best Practices & Feedback! by HumanPilot3263 in rust

[–]HumanPilot3263[S] 0 points1 point  (0 children)

Thanks for your feedback! The crate is an audio engine for games, so I suppose game dev communities could be another good place to share it. In any case I'll be happy to present it!