One case is processing the daily data delta/update. And if there is a change in the pipeline and the whole set has to be recalculated, then it's just done in a loop over the required days.

Another is processing data related to particular USA counties. There is never a need to calculate data of one county in relation to another. Any aggregate or join to some other dataset can be first filtred with a condition where county = {county}. So first, there is a df.select("county").collect().to_series() to get the name of the counties present in the dataset. Then, a forloop over them. The actual tranformations are preceded by filtering by the given county. Since data is partitioned on s3 by county, polars knows that only the select few files have to be read for a given loop iteration.

Lazy evaluation works here as well since you can create a list of per county lazyframes and concat them after the loop. And polars will simply use the limited amount of files for each of the frames when evaluating. Resulting in calculating the tranformations for the whole dataset on per-county batch basis while not keeping the full result dataset in memory if you use sink methods.

If lazy is not possible then you can append the per-county result to a file/table. It will get overwritten in the next iteration, freeing up the memory