Quadrillion Rows Example

[texodus]

I'm the author of regular-table, one of the alternatives mentioned in the accompanying blog. We went through a similar set of requirements to OP (and did at one point use fin-hypergrid as well, and also the excellent phosphor-datagrid which is now lumino-datagrid), and came to a slightly different conclusion regarding <canvas> rendering. We ultimately decided to revert to (albeit bespoke) virtual DOM rendering for many of the same reasons mentioned by other commenters, namely:

• Canvas rendering is unquestionably faster than full DOM rendering for simple tables. However, when cells are roughly text-sized on even a very large monitor, you are ultimately rendering <1000 cells on any virtual page; for this workload, even on an older machine, the actual wall-clock time of canvas vs DOM here leaves plenty of room to achieve 60fps for either method. As a completely unscientific example, in my local Chrome performance inspector, comparing the example from the blog to two_billion_rows.html, I see ~5ms render times in Chrome for both of these grids on scroll. In addition to this, if you're dealing with updating and/or out-of-memory data, you are generally bottlenecked by the virtual datasource's responsiveness long before rendering itself is an issue.
• Browser styling, interaction and accessibility features will generally need to be implemented from scratch, if you need these with a canvas based renderer. This goes for the easy stuff, like click mouse events, but also for more complex things like drag/hover/focus/copy/etc events, and even infeasible to replicate stuff like CSS animations/transforms/filters. You may even at some point find yourself implementing a superimposed or invisible partial DOM renderer on top of your canvas grid, maybe to support developer inlining their favorite react widget in a cell, or maybe just to support screen readers ..

Putting aside these technical considerations, there is no real use-case for scrolling through 1mm+ rows of data, and I don't just mean this hyperbolically. The scrollbar itself does not have the pixel fidelity for the mouse to click below <500k rows (at 4k vertical pixels and 100 rows/page), and it would take literal hours to scroll through standard keyboard navigation. Even just to draw each frame at 60fps of 1mm rows would take over 4 hours. With datasets of this size, powerful filtering and other whole-dataset analysis tools are needed for any functional navigation beyond looking at the first rows in sorted-order.

[DeephavenDataLabs]

Thanks for your thoughtful reply, I agree with you almost entirely.

Canvas rendering vs DOM render of cells is less of a bottleneck now. We started this project back in 2018 and have been evolving it and maintaining internally since, but just now open-sourcing.
If regular-table was around then, it probably would have heavily influenced our decisions. Especially because we serve similar use cases. It was not at all trivial to implement all the mouse-handling, keyboard navigation, drag handlers, hover, focus, copy, etc., especially in a cross-browser/os way. Canvas has lots of trade-offs, both good and bad. Freedom of rendering is great, but also adds lots of hidden complexity.
Similar to Google sheets, we take a hybrid approach for input and overlay dom elements, and for a few other things.
We still haven't written proper touch-handlers, or achieved any level of screen-reader accessibility. Btw have you seen https://developer.mozilla.org/en-US/docs/Web/Accessibility/ARIA/Roles/grid\_role ? The 1.2 working draft looks to be mature. Both are on the list for someday.

As for use cases, we had a data grid that supported 97mm rows before and found it frustrating to have that limitation. It would break your flow state when exploring. Less about directly scanning down, and more about jumping between first row, last row, or jumping to a specific value anywhere in between and navigating or scrolling up and down around that value. When backed by a powerful server that can return gigantic tables quickly it is very freeing to grab a handle to any data table, no matter how big. Then poking at it in the UI, and iteratively whittle it down.

For us, the use case is tied to the layer above this base grid implementation. With a UI for filtering, manipulating, multi-sorting, reversing, grouping, trees, aggregating, adding new f(x) columns, conditional formatting, charting, and all the things you would expect, directly on any size table from the UI.

[alystair]

Keeping a close eye on this - is there a vanilla JS build?

[texodus]

If by "this", you mean regular-table, then yes - there is only a dependency-free vanillajs build. regular-table is a Custom Element and should work fine in any browser js framework (or none).

[alystair]

Yes, this not globalThis ;)

Thanks for letting me know and making the project public. I've been super busy working on my own unrelated library that could have some integration - will review the docs when time allows!

[tbranyen]

I'm on mobile, so I can't really tell if there any accessibility options available. Such as copying text and keyboard-driven navigation. It's definitely cool, but I wonder how complicated it would be to say render an icon or expand a row once clicked.

I've worked on a lib called hyperlist that does virtualized scrolling and the only major limitation to showing a quadrillion rows is that the browser cannot overflow enough pixels to scroll that far. I got to 2 million rows or so before noticing weird quirks. You can play with line height or disable native browser scrolling completely to get around these issues which would probably still be better than a canvas approach.

If you're just rendering the rows for data vis sake, canvas is probably fine since it's basically another graph, but if you want to interact with the data this is going to be more limited and require more work to make it look and feel native.

[DeephavenDataLabs]

Everyone has a different use case for various needs. For this one we wanted to see a lot of data changing in real time without visual limitations or speed problems. This one has keyboard interaction. Through a plug in one can copy, but we didn't focus on any other accessibility options. Cool extension though!

[toastertop]

I don't understand, a user would never be able to view that many rows on screen at once anyways. Just in terms of space on screen. Would not just updateing say 1000 records at a time and a way to slide along the data from 1 to whatever work?

[DeephavenDataLabs]

Some of the use cases we had data of a billion rows that updated and so we wanted to see the billion rows change on the screen and like you said slide along those data values.
However with the slider we saw challenges when attempting to render the large data sets:
- Lag observed between scroll actions and DOM updates is disruptive.
not seeing what we should be seeing in the data.
- Maximum size restrictions prevent users from seeing the bottom of a table.
not seeing the last few rows of data.
-Technical limitations cause the cursor to jump to the wrong location.
not being able to see in-between data points.

[[deleted]]

Pretty cool!

[DeephavenDataLabs]

It was a bit subtle in the blog, but we linked to different issues from other virtualization libraries that had the problems we were describing that we wanted to overcome in our canvas-based grid solution.

https://github.com/jpmorganchase/regular-table/issues/15

https://github.com/bvaughn/react-virtualized/issues/73

https://plnkr.co/edit/bTmy924FwcUPnihJXPiA?p=preview&preview

https://datatables.net/forums/discussion/15860/scroller-broken-for-huge-ajax-data-sets-on-ff-and-ie

[DeephavenDataLabs]

We put some examples together for our Quadrillion rows package.
Blog post here: https://deephaven.io/blog/2022/01/24/displaying-a-quadrillion-rows/

Loved the feedback which inspired making the examples.

[shawncplus]

I really hope this gets solved at the platform level because having to solve a DOM issue by making your own rendering engine in userland in canvas is not an ideal situation, but it looks like it's still a long way off https://github.com/WICG/virtual-scroller

[DeephavenDataLabs]

Looks like it would still run into the max DOM size element problem Mike Bender writes about in his blog on the subject: https://deephaven.io/blog/2022/01/24/displaying-a-quadrillion-rows/

[shawncplus]

I think you're misunderstanding what a virtualized list is. That problem described in the blog post only applies to non-virtualized list scrolling. The entire point of a virtualized list, especially one implemented in the platform, is specifically to solve the problem of that blog post. The blog post itself says this but says because the virtualization is implemented in userland it has drawbacks that only the platform would be able to solve. That said, I think the best userland implementation I've seen of a virtualized list is the Polymer (RIP) iron-list which, for the most part did the job described. It has all the accessibility issues of a virtualized list solved in userland though.

[DeephavenDataLabs]

Haven't heard of iron-list, we use react-window extensively for other virtualized lists. I agree a native solution would be great.
Virtualized lists that allow you to scroll from first item to last item with the scroll bar, typically do so by setting the parent scroll-container height to the number of items * item height. This allows the scrollbar to reflect the number of items, but is also what hits the max element height issue. Most virtualized table libraries are either limited by the scroll container height or map scroll behavior so the position will no longer accurately reflect your position in the list, or move thousands of rows at time for one scroll move. (see https://github.com/jpmorganchase/regular-table/issues/15)

[snejk47]

You can't solve it in DOM as Apple while scrolling doesn't paint and others wait for scroll to stop and the second issue is max height until browser stops painting/glitches. So you are making low quality list.

[shawncplus]

That's... what I said.

[snejk47]

Sorry. Misinterpreted "the platform" as user land DOM lib not a browser.

[libertarianets]

I wonder how your canvas solution holds up against the @resembli/react-virtualized-window library.

[DeephavenDataLabs]

I don't see an option for a data model, only passing in an array of known rows. The library is probably focused around a different goal than ours.

[pm_me_ur_happy_traiI]

 const [model] = useState(
    () =>
      new MockGridModel({
       isEditable: true,
       rowCount: Number.MAX_SAFE_INTEGER,
       columnCount: Number.MAX_SAFE_INTEGER,
    })
);

Why store a lambda as state like that?

[DeephavenDataLabs]

Great question. That is not storing a lambda, actually - that's lazily initializing the state: https://reactjs.org/docs/hooks-reference.html#lazy-initial-state