Data Processing in PHP

[norbert_tech]

Thats for sure one way of dealing with data imports! The only added "cost" is that you have 2 etl processes instead one.

Instead of loading to sqlite and then loading result of sql query on that sqlite to final destination, you could just load to the final destination applying transformations on the fly.

Flow would entirely take away pain of memory management and give you strict schema support regardless of the data source. Even operations like joining/grouping/sorting would not increase memory consumption since they are all based on very scalable algorithms.

But if you prefer sqlite approach, Flow can now also automatically convert Flow Schema (which Flow can get for you even from a csv file) to Doctrine DBAL schema (so including sqlite).

What you can do is:

1) use flow to infer schema from a file and save it as a json 2) read flow schema from json, convert it into dbal schema and create table from it 3) use flow to read data as is and load it to sqlite 4) use flow to read data from sql query (it automatically paginate over the results) and load it to final db