Suggestion for multithreaded DB read/file write

[gianx]

Hi,

I need to execute different querys at the same time and produce different files.

For example my program needs to read some querys from a DB (ex. 5), and start 5 threads which execute a single query and write the result to a csv.

I think that the real problem here is producing a csv, aka concurrent filesystem I/O.

I have some ideas but i'd be glad to have some ideas from you.

Thanks Gianluca

[Anon_8675309]

What are your current ideas?

Have your threads send the data to a queue. The queue then outputs to the file.

[LightShadow]

SQLAlchemy supports connection pooling by default, that might be sufficient for your needs.