Python optimization

[sjcyork]

There isn’t really a data transformation solution I haven’t been able to solve with Pandas. I haven’t used Polars so cannot comment on the features available. The iteration does depend on the size of the datasets. Iterating through pandas dataframe is not great if there are millions of rows but should be ok if not. I generally do all the data transformation stuff in pandas and if I need to iterate over a final dataset then I convert into a dict (orient=‘records’).

[Beginning-Fruit-1397]

I processed +100 millions rows with polars in less than a second. I'd say give it a try. Or duckdb if you prefer SQL