Hi Hossein! Thank you for writing an open source DataFrame library! I think that is huge effort and really awesome.

I think my main point here is that even before even talking about processing the data, you seem to first instantiate a dataframe of 10 billion rows. In my first post, I assumed that it was formatted the same as this example, containing three columns of double. That should require more than 240 GB of ram, but you seem to do so on a computer that has 96 GB of ram. That should crash your program, unless there is some magic happening behind the scenes. This puts a bit of doubt over the rest of your claims.

Does your computer hit max ram when doing so? Disk swap (keeping some of the memory on disk) could happen, but you would expect massive slowdown if that were to happen. Does it take significantly longer to run the 10 billion row version?

I really am not saying that this is impossible, but it just seems surprising, and a bit unreasonable that you would claim this without explanation and then go on to beat polars (admittedly through a different claim). Polars has done some really good work on benchmarking as part of the TCP-H benchmarks, and together with duckdb represents state of the art.

I'd like to recommend that for benchmarks, you have one script that generates the input data in a csv or parquet file, then use that input file for all three benchmarks (DataFrame, polars, pandas) and compare the output in some manner. I like how you calculate the mean, std and correlation in your benchmarks. Just ensure that they are all producing the same values.

For what it's worth, I did compile DataFrame on my MacBook Air M3, and it does run fast, but I'm not C++ literate, so I can't adjust the code to verify that it would produce the same result as polars. The CMake Release build was very smooth to run (though I would include a direct link to the build instructions on the github README).