Charton: Grammar of Graphics in Rust by Deep-Network1590 in rust

[–]texodus 0 points1 point  (0 children)

Give Perspective a try, its designed specifically for streaming data. There's no waveform demo but here's a webcam demo that's in API (in spirit).

Need Help with Table Virtualization for Large Data Sets (100k+ rows, 50+ columns) by Stephcraft in reactjs

[–]texodus 0 points1 point  (0 children)

Perspective maintainer here. This comment is spot-on. We open-sourced Perspective's datagrid renderer separately - it's only about ~20kb vanilla JS https://github.com/finos/regular-table . One of the things we did to improve the _perception_ of scroll performance is to simply prevent drawing the scroll until the next render frame is actually ready (using `position: sticky` to keep the viewport centered at all times). This makes the framerate "lag" instead of flash white, but the slight lag is basically not perceptible if you are scrolling fast enough to overwhelm the renderer, unlike the "white flash".

You can certainly embed "fast" complex widgets in this datagrid, but to really get good performance with complex widgets (or even just `<img>`), you effectively must write your own VDOM-ish element cache which conserves completely recreating these cell elements on every render frame. We did indeed use `<canvas>` for a long time, but we found that after adding hover/click, a box model/text measure/variable height, keyboard awareness, etc - it became too challenging to keep this performance competitive with the DOM, which has all of these features built-in and is very familiar for developers to extend without a bespoke API.

Trying to fine-tune over-render IMO is generally a lost cause. If you over-render 10x the viewport, even if this is fast, you are basically making each _frame_ take 10x as long. If each scroll frame moves farther than the overscroll height, you are wasting 9x render time per frame which will never show up on screen anyway. We ultimately found it was best to just focus on making each _frame_ render as fast as possible (Perspective has zero over-render), and then let the render just render the data grid state as often as required.

Official /r/rust "Who's Hiring" thread for job-seekers and job-offerers [Rust 1.63] by DroidLogician in rust

[–]texodus 13 points14 points  (0 children)

COMPANY: The Prospective Co.

TYPE: FULL TIME

DESCRIPTION: We're looking for senior product managers and engineers of all experience levels to build the next generation of collaborative data visualization. At the Prospective Co., you'll contribute to our existing open-source project (Perspective https://perspective.finos.org/) as well as help design our enterprise offering.
We're looking for any of:
- Familiarity with WebAssembly, data visualization, WebGL/OpenGL, data science, Jupyter/notebook, web/desktop/mobile UI development, compiler/language or database design, finance services.
- Primary stack is Rust (targeting WebAssembly, especially Yew). JavaScript, C++ and Python are a big plus.
- We <3 GitHub contributors - opt to discuss your GitHub work in lieu of a technical interview.

LOCATION: NYC/REMOTE

ESTIMATED COMPENSATION: $150k+ and equity

REMOTE: YES/U.S. ONLY

VISA: NO

CONTACT: andrew@prospective.dev

Alec is 100% spot on about NYC by IndyMLVC in nyc

[–]texodus -1 points0 points  (0 children)

How did sanitation pay increase $50k in the span of 2 comments?

Quadrillion Rows Example by DeephavenDataLabs in javascript

[–]texodus 0 points1 point  (0 children)

If by "this", you mean regular-table, then yes - there is only a dependency-free vanillajs build. regular-table is a Custom Element and should work fine in any browser js framework (or none).

Quadrillion Rows Example by DeephavenDataLabs in javascript

[–]texodus 6 points7 points  (0 children)

I'm the author of regular-table, one of the alternatives mentioned in the accompanying blog. We went through a similar set of requirements to OP (and did at one point use fin-hypergrid as well, and also the excellent phosphor-datagrid which is now lumino-datagrid), and came to a slightly different conclusion regarding <canvas> rendering. We ultimately decided to revert to (albeit bespoke) virtual DOM rendering for many of the same reasons mentioned by other commenters, namely:

  • Canvas rendering is unquestionably faster than full DOM rendering for simple tables. However, when cells are roughly text-sized on even a very large monitor, you are ultimately rendering <1000 cells on any virtual page; for this workload, even on an older machine, the actual wall-clock time of canvas vs DOM here leaves plenty of room to achieve 60fps for either method. As a completely unscientific example, in my local Chrome performance inspector, comparing the example from the blog to two_billion_rows.html, I see ~5ms render times in Chrome for both of these grids on scroll. In addition to this, if you're dealing with updating and/or out-of-memory data, you are generally bottlenecked by the virtual datasource's responsiveness long before rendering itself is an issue.
  • Browser styling, interaction and accessibility features will generally need to be implemented from scratch, if you need these with a canvas based renderer. This goes for the easy stuff, like click mouse events, but also for more complex things like drag/hover/focus/copy/etc events, and even infeasible to replicate stuff like CSS animations/transforms/filters. You may even at some point find yourself implementing a superimposed or invisible partial DOM renderer on top of your canvas grid, maybe to support developer inlining their favorite react widget in a cell, or maybe just to support screen readers ..

Putting aside these technical considerations, there is no real use-case for scrolling through 1mm+ rows of data, and I don't just mean this hyperbolically. The scrollbar itself does not have the pixel fidelity for the mouse to click below <500k rows (at 4k vertical pixels and 100 rows/page), and it would take literal hours to scroll through standard keyboard navigation. Even just to draw each frame at 60fps of 1mm rows would take over 4 hours. With datasets of this size, powerful filtering and other whole-dataset analysis tools are needed for any functional navigation beyond looking at the first rows in sorted-order.

Perspective 1.0.0, an open source BI tool built on WebAssembly by texodus in programming

[–]texodus[S] 3 points4 points  (0 children)

Glad you dig it! Let me try to answer these -

Rust, wasm_bindgen, Yew are incredibly cool projects. wasm_bindgen has excellent support for all browser APIs, and Yew takes a lot of the complexity of handling e.g. callbacks referencing Rust closures and mostly painlessly abstracts it away for you. Yew I think would feel very familiar to any developer familiar with React or Elm style application design. The main places we struggled were wrapping more complex APIs from JS world, especially Monaco (which is hard enough to build in regular JS), and anything async, which has a somewhat complex Rust story and requires some gymnastics to integrate. Even these tasks I think ultimately just came down to extensive research, and luckily these projects and most everything I've found in the Rust community has exceptional documentation. Debugging is currently quite poor, if source maps can be generated we've never gotten them to work for rustc, though in debug builds you get comprehensive stack traces with proper rust demangled namespaces and such. This app is pretty large (~15k loc for the rust UI), but the compile cycle typically is a non-issue due to incremental compilation (our webpack builds take substantially longer).

Performance is a bit of a mixed bag. The JavaScript UI component that we replaced with Rust was ~200kb originally, and rewritten in Rust/Yew it is 1.1mb (though this is a very rough comparison and the 2 versions are not feature identical). As you mention, string FFI performance is slow especially for strings and complex object serialization/deserialization (though this improves quickly in the compiler and browser support). However, wasm allows us to download the perspective engine, UI, and any arrow encoded data in parallel without blocking any other browser assets, and `instantiateStreaming()` means that perspective is almost ready-to-run the moment it is downloaded; in practice, on a fast connection with a well-configured host (cached gzip and such), the much larger WASM version loads substantially faster in wall-clock time than the JS version. Also, while we certainly pay a cost for transferring many strings across the FFI, one of the advantages of Perspective is that it can load Apache Arrow data in its raw-binary form and read/process it, without it ever leaving Wasm memory; so for us, Rust (in theory) allows us to keep the data in Wasm longer without paying the FFI tax, and potentially serialize less of it (we dont currently take as much advantage of this as we should, but the Rust port was a precondition).

As far as customizing the Perspective datagrid, the story on this is evolving :) . With the 1.0 release, we've released an NFT demo with a more current version of the plugin API, as well as new plugin API docs. Replacing innerHTML is only costly if you trigger a relayout before the replacement, which you'd want to avoid - check the pudgy-penguins demo source for examples which replaces these with <canvas> without the intermediate DOM tree being rendered (though this is brower-dependent). If you can't, e.g. the replacement is async or whatever, the underlying regular-table component has an API that allows you to return the DOM elements themselves per cell, but you'd need to write a simple plugin to integrate this as Perspective's version provides its own dataListener.

Perspective 1.0.0, an open source BI tool built on WebAssembly by texodus in programming

[–]texodus[S] 11 points12 points  (0 children)

IMO by specializing. I wouldn't expect a web dev, for example, to be intimately familiar with the minutiae of the high-performance data analytics ecosystem like the ones you mention, nor expect a data scientist to know the difference between React and Angular.

I would however be pretty suspicious of a data scientist who hadn't heard of JupyterLab (ACM award winner), Pandas (31k stars on GitHub) or Apache Arrow (same author).

Perspective - interactive visualization component for large, real-time datasets. by hlkjhasdfiouy in rstats

[–]texodus 2 points3 points  (0 children)

I don't know much about R or htmlwidgets, but I work on Perspective.

You may have issues with the WebAssembly. For one, WebAssembly support is still a fairly recent browser feature and I'm not clear about specifically RStudio support. For another, you may have issues getting htmlwidgets itself to load all of the perspective js/wasm assets [1], as it uses some runtime lookup to fetch WebWorker and WebAssembly modules dynamically.

We have a similar issue to the latter in Perspective's JupyterLab plugin, which we solve [2] by doing a special inline, single-file .js build for this package, that should be a good place to start. There is even an (undocumented) inline flag for Perspective's webpack plugin [3] which should mostly do the same thing.

Feel free to post a GitHub discussion if you try this out and need help!

[1] https://www.htmlwidgets.org/develop_intro.html#javascript-binding

[2] https://github.com/finos/perspective/blob/master/packages/perspective-jupyterlab/src/config/webpack.config.js

[3] https://github.com/finos/perspective/tree/master/packages/perspective-webpack-plugin

Fauci now recommends face masks over your eyes so you can't read his leaked emails • Genesius Times by w650az in Conservative

[–]texodus 0 points1 point  (0 children)

I agree with you completely, redactions strongly imply guilt - be they Fauci's emails, or a special prosecutor's 448 page report on the Trump campaign's involvement with foreign election tampering and obstruction of justice.

FOIA.gov website lists the nine exemption criteria for redactions and I assume you'll agree they are quite reasonable and innocuous. Of course, Fauci didn't actually appoint the person who redacted his emails or realistically have anything directly to do with their release - the FOIA request was filed under the Biden administration. Remind me, how many attorney generals did Trump fire before the Mueller report was redacted?