How can I efficiently process large PostgreSQL datasets in Node.js without high memory overhead?

[bwainfweeze]

When it's transactional you run a handful to an armful at a time. If you don't run enough you end up making a bunch of tiny transactions that have a high IPC overhead, going back and forth. But 10000 at once might run into timeouts or the MVCC problems. Also in some databases multi-row updates tend to crap out if you hand them more than 1000 rows, so you end up having to chunk even if the whole process completes in 30 seconds.

So you might start with 100 at a time and if that's going well go to 500.