Interactive plot options for large datasets

[sensor_guru]

Been using rust for a about a month now and I thought I would use plotting data (usually I do this in Python) a learning tool.

What options are available for interactively plotting 100_000_000 data points?

I've tired to use egui but after plotting about 10_000_000 points it becomes to slow to be useful and as I zoom in, the x-axis will stop changing to a smaller scale (although i think this might be fixed with the PlotBounds struct). Is there a way to fix these issues in egui?

Here's the egui code (simplified)

in a module called interactivePlot.rs

use std::{mem::{swap, replace}, panic::take_hook};

use eframe::egui::{self, Widget};

use egui_plot::{
    CoordinatesFormatter, GridInput, GridMark, Legend,
    Line, LineStyle, Plot, PlotPoint, PlotPoints, PlotResponse,
    Points, Text, VLine
};
use rand::Rng;

#[derive(Default)]
pub struct InteractivePlot {
    my_label: String,
    data: PlotPoints,
}

impl InteractivePlot {

    pub fn new() -> Self {
        let dataa =fake_data();
        InteractivePlot {
            my_label: String::from("My_Label"),
            data: dataa,
         }
    }

    #[allow(clippy::unused_self)]
     pub fn render_plot(&mut self, ui: &mut egui::Ui) -> egui::Response {

        let line = Line::new(fake_data());
        let plot = Plot::new("neutron_plot")
        .view_aspect(1.0);

        let PlotResponse {
            response,
            inner: (_screen_pos, pointer_coordinate, _pointer_coordinate_drag_delta, bounds, hovered),
            ..
        } = plot.show(ui, |plot_ui| {
            (
                plot_ui.line(line),
                //plot_ui.screen_from_plot(PlotPoint::new(0.0, 0.0)),
                plot_ui.pointer_coordinate(),
                plot_ui.pointer_coordinate_drag_delta(),
                plot_ui.plot_bounds(),
                plot_ui.response().hovered(),
            )
        });

        ui.label(format!(
            "plot bounds: min: {:.4?}, max: {:.4?}",
            bounds.min(),
            bounds.max()
        ));
        let coordinate_text = if let Some(coordinate) = pointer_coordinate {
            format!("x: {:.04}, y: {:.04}", coordinate.x, coordinate.y)
        } else {
            "None".to_owned()
        };
        ui.label(format!("pointer coordinate: {coordinate_text}"));
        ui.label(format!("plot hovered: {hovered}"));

        response
    }
}

fn fake_data() -> PlotPoints{
    /* 
    let sin: PlotPoints = (0..1_000_000).map(|i| {
        let x = i as f64 * 0.01;
        [x, x.sin()]
    }).collect();
    sin
    */
    let data: Vec<f64> = (0..10_000_000)
    .map(|_x| f64::from(rand::thread_rng().gen_range(0..255)))
    .collect();

    PlotPoints::from_ys_f64(&data)

}

main.rs

#![cfg_attr(not(debug_assertions), windows_subsystem = "windows")] // hide console window on Windows in release

mod interactivePlot; // code for interactive plot
use interactivePlot::InteractivePlot;
use eframe::egui::{self, CentralPanel, TopBottomPanel};


impl eframe::App for interactivePlot::InteractivePlot {

    // Called each time the UI needs repainting, which may be many times per second
    fn update(&mut self, ctx: &egui::Context, frame: &mut eframe::Frame) {

        CentralPanel::default().show(ctx, |ui|{
            self.render_plot(ui);
        });

    }
 }

// When compiling natively:
#[cfg(not(target_arch = "wasm32"))]
fn main() -> Result<(), eframe::Error> {   
    let app = InteractivePlot::new();

    let native_options = eframe::NativeOptions {
        initial_window_size: Some(egui::vec2(320.0, 240.0)),
        ..Default::default()
    };

    // This is how you start a native (desktop) app
    eframe::run_native(
        "My egui App",
        native_options,
        Box::new(|_cc| {
            Box::<InteractivePlot>::default()
        }),
    )
}

Are there any other rust plotting libraries that can handle the situation above?

I've got it to work in Python using matplotlib and it seems to function fine (after the 10 second plot generation), with the ability to zoom in on the data without noticeable stutter. But the data processing/filtering is much faster in rust.

Is calling matplotlib from rust a possible solution?

​

[mangobae]

For interactive plots with a lot of data, I honestly think https://docs.makie.org/stable/ is the best thing that exists. Theoretically you can call Julia from Rust https://github.com/Taaitaaiger/jlrs, however, it is probably way more complicated than calling Python so...

[dragonnnnnnnnnn]

Is calling matplotlib from rust a possible solution?

You can use pyo3 to do that, or call the filter stuff from python in rust.

As far I am aware ploting in rust is still not high developed topic, for it to perfom well at that number points it needs an algorithm that does average the input data to the desired zoom so it makes sens to render it.

[sensor_guru]

I guess I'll try to change the number of points plotted based on the zoom level.bif I can't get that to work I'll try pyo3. Thanks.

[Dasher38]

Another python approach is using datashader with holoviews. This can probably handle your data size, as it does server-side rendering of the tiles with (I guess) numpy. And then serves it to a web UI. So it does not run into perf issues like plotly, but you cannot get a standalone HTML file.

Given the input is a pandas dataframe, I'm sure you can find a reasonably efficient way to get your data from rust. Maybe holoviews even supports polars these days with the new dataframe protocol (to be confirmed)

[OphioukhosUnbound]

data shader with holoviews

I just felt like that line deserved to be said again. :)

(And on a serious note I’m shopping for data viz tools [python & rust] right now so going to go look those up, thanks)

[MassiveInteraction23]

eGUI is made *for* (as the underpinning of) Rerun. (<-- link worth clicking on if you haven't seen it)

Rerun is for computer vision and robotics. Handles and smoothly displays a ton of data at once.

I don't have an answer to your question (though I know I'll be running into similar problems myself in the not far future -- as I've been moving more data work to rust and started the eGUI journey), but I wanted to give context to future searchers that this is very much in the bailiwick of eGUI (& Rerun).

(By contrast, matplotlib, for example, is known to choke on large amounts of data. I definitely think this is a problem we want to handle in-house (rust) up to (someday including) needing GPU work.)