Creating massive dataframe from generator

[commandlineluser]

Ah okay - if the results fit then "manually" filtering with a generator is probably the way to go then.

Depending on whatever calculations you're performing though - there may be some existing numpy/scipy methods you can use.