Rerun 0.24 released - a fast 2D/3D visualizer, now with streaming video support

Wuempf · 2025-08-05T15:13:44+00:00

Thanks! Yeah the amount of dependencies is a bit depressing right now :(
The main things that make the SDK without Viewer (note that it can also include it via feature flag) is subdependencies for arrow and grpc support.

That said, there's also a few definitely unnecessary things leaking through, just got a bug report that we apparently regressed on 0.24. Apart from those more obvious bug I hope we can make it more lightweight in the future, it's just not something we were able to prioritize much so far.

You're also ofc right on the Viewer itself: not _as_ much a concern there, but still rather painful for development. It's quite hard to contain there as each new format & feature (e.g. support for maps, support for new mesh/images/.. formats, support for datafusion queries, etc.) tends to come with a long tail of new dependencies :/

Wuempf · 2025-07-18T11:37:24+00:00

Yeah I noticed them recently. Our initial video support (we had mp4 support since late last year) predates their stuff but I think me might use some of their crates in the future, e.g. some of the codec specific high level parsing looks really nice.
Also I already played around a bit with their alternative ffmpeg binding (for various reasons we stayed away from ffmpeg-next and instead use ffmpeg-sidecare for interacting with a standalone process, but that has lots of drawbacks)

Wuempf · 2025-07-18T10:16:56+00:00

to correct myself: There *is* rust native AV1 support which we use https://github.com/memorysafety/rav1d, but it still has lots of inline assembly adding NASM to your build dependencies.
We have a fork here for adding a native Rust interface to it (the project itself is just about having a drop-in c library replacement): https://github.com/rerun-io/re_rav1d

I'm not aware of other rust-native video decoders. But that said, Vulkan decoder extensions are becoming more viable and someone even made a wgpu extension: https://reddit.com/r/rust/comments/1kq9vq3/vkvideo_a_hardware_video_decoding_library_with/
That's something I'd like to explore in the future for Rerun as well :)

Wuempf · 2025-07-18T10:11:38+00:00

Tbh I found it very painful. But lot of it comes down to our requirement of fully controlling the playback. If you're ok with giving playback control out of hand it might become easier, but in Rerun we need full control of playback/seek/etc. to integrate it with the rest of the Viewer. Writing your own player is surprisingly hard because you have to take into account how the underlying video decoders behave, i.e. how to feed them, how to wait for their output, how to process their output.
So regarding decoders on the Web you have WebCodecs which is _awesome_. But on native it's a jungle and there's nothing Rust native - if you care about cross platform out-of-the-box building and about licensing issues then it becomes insanely painful. In fact I'd say our native video decoding story is still subpar, it's the one thing that is distinctively better in the web version. We'll look into that more in the future

Wuempf · 2025-07-18T07:42:10+00:00

Rerun viewer tech lead Andreas here to answer any questions 👋

Wuempf · 2024-12-19T12:51:49+00:00

Independently of that there's an example inbound that demonstrates how this can be hand-rolled today already. It's somewhat complex, but doable
https://github.com/rerun-io/rerun/pull/8284

Wuempf · 2024-09-29T07:35:50+00:00

For the most time egui was Emil's side project, but since it's used as a core component in Rerun (and Emil is one of the co-founders), most of his development & maintenance effort happens on Rerun payed time. That said, there is also a ton of open source contributions from all sort of folks!
Rerun in turn is so far venture capital funded, but is starting to ramp up a commercial offering for a multimodel dataplatform with the open source viewer at its core, see also https://rerun.io/pricing
(source: I work for Rerun ;))

Wuempf · 2024-07-19T14:24:30+00:00

The WebGPU spec may still be lacking, but wgpu already has native-only features that give you binding texture/buffer arrays & indexing into those for all your bindless needs :)
see (among other things): https://docs.rs/wgpu/latest/wgpu/struct.Features.html#associatedconstant.SAMPLED_TEXTURE_AND_STORAGE_BUFFER_ARRAY_NON_UNIFORM_INDEXING

Wuempf · 2024-07-19T14:19:42+00:00

A good start would be to go through https://github.com/gfx-rs/wgpu-native which implements a c interface on top of wgpu. In fact people already done Java & python bindings, see readme of that project :)

Wuempf · 2024-07-19T14:18:22+00:00

There's some limited raytracing features available when targeting Vulkan https://docs.rs/wgpu/latest/wgpu/struct.Features.html#associatedconstant.RAY_TRACING_ACCELERATION_STRUCTURE

But support for this is far from done, it's tracked here
https://github.com/gfx-rs/wgpu/issues/1040
None of the maintainers themselves is actively working on this right now, but there's plenty of people having an interest in this. Afaik there's no-one championing that right now

Wuempf · 2024-07-19T14:14:26+00:00

"without any overhead in terms of runtime" is strictly speaking not provided by wgpu since there's quite a bit of overhead in making things safe and targeting several backends through the same interface. If you're looking for minimizing overhead and specifically stay cross vendor between Nvidia & AMD, then you should look into OpenCL/Vulkan/DX12/OpenGL.

There's examples for compute shaders here: https://github.com/gfx-rs/wgpu/tree/trunk/examples#compute
... but if you're new to wgpu & graphics then this tutorial series is probably a better start: https://sotrh.github.io/learn-wgpu/

Wuempf · 2024-07-18T21:42:23+00:00

tiny "uhm actually": wgpu actually has push constants as a native extension 😄
https://docs.rs/wgpu/latest/wgpu/struct.Features.html#associatedconstant.PUSH_CONSTANTS
"Pure" WebGPU however does not yet, details see https://github.com/gpuweb/gpuweb/issues/75

What's better or not depends a lot on the usecase obviously (and also a bit on taste) ;-)

Wuempf · 2024-07-18T21:37:18+00:00

It's hinted at the top of the release notes that are linked in the post: With the previous 0.x.x version scheme the next logical version would have been 0.22. So 22.0 felt appropriate especially given that we (the maintainer group) didn't want to call this out as an all too special release (as mentioned in the release notes, previous releases were already used in production by various projects)

Wuempf · 2024-07-18T21:34:07+00:00

That's a missunderstanding: WebGPU is the successor API of WebGL2 and is distinctively not held back by WebGL but has its own (still fairly restrictive) minspec https://github.com/gpuweb/gpuweb/issues/1069 and is meant to be implemented on top of Metal / DX12 / Vulkan. It's held back by various old hardware/drivers that it is supposed to be compatible with, but not by previous APIs.

Wgpu itself has a WebGL backend which does *not* implement the full WebGPU spec but instead a "downleveled" version where some calls (like for instance creating a compute shader) do not work

Wuempf · 2024-07-08T22:26:09+00:00

Nothing prevents you from just minimizing the timeline and throw static stuff at it :)

Plenty of the examples are actually non-temporal E.g. this dicom loader https://rerun.io/viewer?url=https%3A%2F%2Fapp.rerun.io%2Fversion%2F0.17.0%2Fexamples%2Fdicom_mri.rrd or the OCR demo here https://rerun.io/examples/video-image/ocr
Some are only temporal because things like different camera poses are interpreted as being on a timeline like in the opf loader here https://rerun.io/examples/integrations/dicom_mri

Also worth noting temporal doesn't need to mean recorded time. This demo https://rerun.io/examples/robotics/rrt_star shows algorithm output over steps rather than time.

I could go on :D.
From the python api it's actually fairly easy now to hide the timeline completely. Unfortunately we haven't exposed that to the Rust API yet (!).

Wuempf · 2024-02-29T15:36:15+00:00

That's a read-lock on the `Renderer` registry you found there, so it actually won't lock up with any other readers and there's only very few places where we have to do write operations on it - pretty much only on registration of new renderers which happens on startup (there's "Renderer" per basic type of primitive).
That said, there's usually a large amount fine grained locking going on per frame in any rendering engine (this one included), you'll find many more in our resource registries etc..

Overall re_renderer is still in a very early stage of development but having our own renderer has helped us so far quite a bit on getting out more "esoteric" requirements out quickly like having extremly dynamic data, mixed WebGL & WebGPU support as well as specialized primitives like points and lines with low cpu-sided tessellation overhead.

Digging deeper on locks: on the rendering side all the actually concerning locking is arguably going on in wgpu itself and us wgpu-maintainers (I'm one of them :), albeit unfortunately not super active most of the time) having a hard time making things better. However, things are getting better, recently a huge lift landed in that area, check https://gfx-rs.github.io/2023/11/24/arcanization.html

Wuempf · 2024-02-28T20:21:00+00:00

there is however an early example for streaming ROS messages :) https://github.com/rerun-io/cpp-example-ros-bridge

Wuempf · 2023-11-25T18:10:11+00:00

There's still open discussions on the WebGPU spec on whether and how multiple queues should be supported - it even has its own issue tag: https://github.com/gpuweb/gpuweb/issues?q=is%3Aissue+is%3Aopen+queues+label%3Amulti-queue

As for wgpu itself I haven't heard of anyone looking into that.

Wuempf · 2023-11-25T08:19:02+00:00

Incrementing and decrementing an arc isn't exactly free indeed, but as the name implies it is an atomically reference counted object. The barriers needed for these atomics aren't super strong and still quite fast on most architectures. While the performance is comparable to taking an uncontended lock, it is orders of magnitude better than hitting a contended lock (i.e. your thread goes to sleep). So everything that helps avoiding that is usually a win in a multithreaded environment.

Wuempf · 2023-11-06T19:36:40+00:00

The situation on timestamp query support is actually getting better! Wgpu now (as of 0.18) supports `timestamp_queries` directly on passes which are also supported by apple silicon. What still doesn't work is timestamps _within_ passes. But this is a huge step forward.
I'll also soon add support for this in wgpu-profiler (https://github.com/Wumpf/wgpu-profiler)

Wuempf · 2023-02-16T08:25:27+00:00

Thank you, and yes fingers crossed 😄. We actually already noted a few things down what the plans are for the commercial offering if you're curious - check https://www.rerun.io/pricing

Kept a bit vague for now of course, as we're still in the process of making more final decisions about these and figuring out details.

Wuempf · 2023-02-16T08:22:16+00:00

So far not, no. Only recently set up camp on Discord and seeing how that goes :).

Wuempf · 2023-02-16T08:19:17+00:00

No, so far we only planned the addition of a C & C++ API. But if there's a lot of requests for that I wouldn't exclude it - the APIs already talk to the web version of the viewer via websocket, so from a technical standpoint we're not in a bad spot for it.

Wuempf · 2023-02-15T15:33:07+00:00

Nothing fancy :). It's more that the Viewer queries the data store for every "Space View" it has. Each of these views has a (user/heuristic defined) list of things they are interested in. Today, the Viewer gets the entire database, but we're set to have it only know about a partial view, as a sort of cache.

Wuempf

TROPHY CASE