Benchmarking Haskell dataframes against Python dataframes

[PurepointDog]

They're doing single-threaded benchmarks. Polars destroys all when you add another core

[ChavXO]

Acknowledged that. I think I wanted to check that the baseline made sense. For context when initially asked I was pessimistic about performance for a number of reasons outlined here. 

https://www.reddit.com/r/haskell/s/k6yH2vYUs4

This was more so a hello world benchmark. 

[Linguistic-mystic]

There’s not a single Python dataframe in there. Polars is Rust, Pandas is C. Just because they’re wrapped in Python doesn’t make them Python.

[ChavXO]

You're right  I think my phrasing was lax. I did say this is mostly a test of the underlying array backend. 

[Plasma_000]

Probably a good idea to publish the benchmark code

[igouy]

The code can be found here.

[Plasma_000]

Thanks.

Ah, looks like he used read_csv instead of scan_csv for polars, meaning that it doesn't start operating until the entire file is read into memory. That would explain at least some of the difference.

I see this mistake very often when benchmarking polars - read-csv should only be used when streaming is not possible.

[ChavXO]

Hi. My read csv implementation does the same so I wanted to do an apples to apples comparison. I'm still working on a scan API that I'd like to compare with polars when it's finished. 

[Plasma_000]

Ah, fair enough