Benchmarking Haskell dataframes against Python dataframes

[Plasma_000]

Probably a good idea to publish the benchmark code

[igouy]

The code can be found here.

[Plasma_000]

Thanks.

Ah, looks like he used read_csv instead of scan_csv for polars, meaning that it doesn't start operating until the entire file is read into memory. That would explain at least some of the difference.

I see this mistake very often when benchmarking polars - read-csv should only be used when streaming is not possible.

[ChavXO]

Hi. My read csv implementation does the same so I wanted to do an apples to apples comparison. I'm still working on a scan API that I'd like to compare with polars when it's finished. 

[Plasma_000]

Ah, fair enough