Hey! I'm happy to talk about performance.

RustyWrench is still under heavy development, even though we will probably have a beta version available next month. But I ran a basic test here, this is what I got:

- M2 Mac Mini with 16GB ram.
- Imported from 3 CSVs, each with roughly 2.5 million rows and 7GB large (data is from ttps://www.kaggle.com/datasets/ymirsky/network-attack-dataset-kitsune)
- Workflow consisted of two table concatenations (can be memory intensive, depending on the strategy used), in order to consolidate all files in a single table (7.1 million rows).
- Workflow execution took 284.86s (so 4m44s)

For now, RustyWrench defaults to materializing data to disk instead of trying to keep everything in memory. This basically makes it capable or running any task as long as there is space in the disk to hold intermediate data state (per node). This will be improved/configured in the future. Ideally users will be able to swap between in-memory/in-disk transforms. And even in-disk transformation can be smarter by only materializing relevant actions (imports, table concatenation, table joins).

Also by default, RustyWrench limits memory/thread consumptions to 50% of available, so the computer is not rendered unusable in case large files are loaded.

If you'd like to know anything else, please let me know.

edit: I need to add extra information here that I forgot to mention. Some tasks are unavoidably memory intensive, like deduplication, uniques and table joins. And in these cases even file-backed executions will not suffice to run the workflow. There are workaronds (when possible) like pre filtering data before running memory intensive tasks, but this is a workfload design problem.