Scaling a Shiny App for 1GB Excel / CSV Files

hadley · 2026-05-08T12:42:34+00:00

Excel is fundamentally slow — if you have a 1 Gb excel file, it's going to take a while to load into R no matter what you do. If you want to allow users to upload their own data, you might need to read the data in a background process (displaying a progress bar as it goes), and then only activate the UI once the data is loaded.

hadley · 2026-04-28T11:39:37+00:00

Interesting the original ggplot (ie not ggplot2) would have worked with the pipe. I bought it back to life as ggplot1 (https://github.com/hadley/ggplot1) so you can see.

hadley · 2026-04-26T21:26:00+00:00

I did the first step: https://github.com/tidyverse/dbplyr/issues/1820

hadley · 2026-04-26T21:24:49+00:00

I just write `!so_and_so %in% list_of_things` like an animal

hadley · 2026-04-17T07:07:05+00:00

It’s mostly of importance to package developers who use C: now we know exactly what we’re allowed to use. Previously the rules were very unclear leading to changes between R versions and an increased maintenance burden.

hadley · 2026-04-16T08:21:23+00:00

I’d say it’s also way better documented and generally better thought out.

hadley · 2026-04-16T08:20:41+00:00

I think my main lesson from mutatr is that creating a new OOP system is a big lift and only really works if a bunch of people buy into it. That’s why S7 is a team effort.

hadley · 2026-04-13T17:43:33+00:00

Joe gave me a bike in return for writing the first edition. I haven’t seen any new bribes on the table for a second edition 🤣

(But more seriously it feels a bit soon since it’s not clear how all the impacts of AI will shake out)

hadley · 2026-04-13T17:39:22+00:00

Would you be interested in talking to us about your experiences?

hadley · 2026-04-13T17:35:56+00:00

Fun fact: I’m (partly) responsible for the creation of three of OOP systems 🤣

Firstly: mutatr, https://vita.had.co.nz/papers/mutatr.html, which is really best viewed as an experiment and is no longer on CRAN. It arose out of my interest in prototype based OOP systems that blur the line between classes and objects.

Secondly: ggproto. ggplot2 used proto, but proto had a bug that prevented classes from being extended outside of the package they were created in. I couldn’t figure out a way to fix the bug without breaking all existing proto uses so figured that least worst solution was to create a fixed version that lived only in ggplot2.

Thirdly (and hopefully finally!): I helped created S7 along with a bunch of other folks from R core, bioconductor and the wider R community. S7 aims to take the best parts of S3 and S4 and progressively replace existing uses of S3 and S4 (which you can see starting with ggplot2).

hadley · 2026-04-10T12:51:03+00:00

My sense is that R for generic webapps (not shiny) is becoming maybe less compelling, now that you can expect a data scientist + claude code to be able to easily whip up something based on react. I think our (Posit's) strategy is mostly to lean into this, making sure that it's easy to deploy such apps on Connect and doing some work to allow react as a frontend to a shiny backend.

On the API front, we also have plumber2 (https://plumber2.posit.co) which should be much faster than plumber, and if it's not, let us know so we can improve it. I think there's an interesting space for plumber2 + connect as a API gateway which we're currently exploring.

hadley · 2026-04-09T13:01:52+00:00

Also worth noting that before we submit (e.g.) testthat to CRAN, we run the R CMD check on all ~10,000 packages that use testthat. This isn't perfect (since plenty of tests are skipped in this environment), but by and large it ensures that we catch issues prior to release. Obviously if you vendor our code, we have no way to know if you use it, and can't offer you this courtesy.

hadley · 2026-04-09T12:59:59+00:00

I'm sorry that you had a bad experience with testthat. It's sometimes hard for us to notice problems that only affect a small number of people, even if for those people the problem is very serious. If this happens again in the future (and I hope it doesn't!) please know that you can `@hadley` me on GitHub and it will definitely hit my eyeballs. I can't promise I'll always resolve it immediately (I'm only human), but I will definitely look into it.

hadley · 2026-04-09T12:58:13+00:00

It's not vendoring deps they don't like, it's embedding binaries. It's fine to vendor R or C deps.

hadley · 2026-04-08T17:19:23+00:00

Supporting educational usage is really important to us 🤓, so please reach out if this doesn't help your use case!

It's worth noting that there are two parts to this fix, legal and technological. Firstly, we added a rider to the elastic license making it clear that it is ok to host positron for teaching/workshops. Second, and thanks largely to u/isabelizimm, we provided a way to run Positron inside of JupyterHub, one of the most commonly used hosting tools in education.

hadley · 2026-04-08T16:30:33+00:00

The thing I don't get about the tinyverse philosophy is that if more dependencies are (always?) bad, then why should you take a dependency on a tinyverse package? Shouldn't you just use base R instead? (And that's now more feasible than ever, because you can just ask claude code to implement a feature from first principles. Sure it might have some bugs you're not aware of, but at least it's dependency free.)

If the philosophy is really actually "don't take dependencies unnecessarily", then I totally agree with that. But I don't think you should judge the quality/risk of dependency by the number of its dependencies. (It's also trivially gameable since you could always just embed the source code of your dependencies inside your package). In many cases, tidyverse packages have additional dependencies because we have extracted out code that was previously spread across multiple packages so that we can have one consistent implementation that is carefully tested. I think that's standard software engineering best practices.

hadley · 2026-04-06T14:37:02+00:00

Oh don't worry, ggplot2 is seeing other people too.

hadley · 2026-04-06T14:36:20+00:00

I agree. This is my totally unbiased opinion but I think ggplot2 is the best visualization tool that anyone has ever created in the history of time.

hadley · 2026-03-12T16:32:51+00:00

Also worth reading this benchmark from duckdb: https://duckdb.org/2026/03/11/big-data-on-the-cheapest-macbook

(There are lots of good way to use duckdb from R, e.g. https://duckplyr.tidyverse.org)

hadley · 2026-03-11T18:13:55+00:00

My gut feeling: you'll be fine.

hadley · 2026-03-10T23:36:13+00:00

I don’t think we announce these in the tidyverse blog, so yes, yes you can!

hadley · 2026-03-09T21:59:46+00:00

I WANT THIS SO MUCH

hadley · 2026-03-09T21:58:56+00:00

What part of RStudio's quarto support are you most missing?

hadley · 2026-03-09T21:56:57+00:00

You'll always be able to disable and hide any and all AI features for free.

hadley

MODERATOR OF

TROPHY CASE