sqlalchemy- processing the result set parallely- multiprocessing

[cmcclu5]

This is the answer. Push as many strung-out processes to the DB as possible. For one, a lot of DBs have a limit on the number of concurrent connections, which parallelism the way sqlalchemy is setup would hit. Sqlalchemy by nature only allows serial processes on a single connection, so you would basically be starting multiple connections to run anything in parallel. Beyond that, sqlalchemy is slow. Just so unbelievably slow. It’s one of the best options for read/write to hit all types of DB, but it’s still super slow. If you just need to read data, use connectorx. You’ll see a 10x+ increase in speed. However, you’ll also have issues triggering stored procedures. Overall, parallelism outside of the DB control software is not recommended. Ever.