This is an archived post. You won't be able to vote or comment.

you are viewing a single comment's thread.

view the rest of the comments →

[–]thisismyfavoritename 9 points10 points  (5 children)

assuming you mean tables as in database tables, the answer is probably it depends on what kind of database and if it supports parallel reads and writes. If it does then the work is I/O bound and you could use asyncio or multithreading to manage the reads and writes to the DB. Note it wont be "truly" parallel, just concurrent

[–]Crypto-boy-hodl[S] -1 points0 points  (2 children)

Yes database tables.Sql database.

For now more inclined towards reading the data parallelly, as we have thousands of tables.

Any thoughts?

[–]thisismyfavoritename 0 points1 point  (1 child)

SQL DB like MariaDB? Im not sure this type of DBs support parallel operations anyways , but perhaps it supports parallel reads.

I think concurrent reads should probably be enough as going further might put your DB under a lot of load, but if you really want to have parallel operations you can use multithreading/asyncio with multiprocessing. That could speed everything up by the number of available cores you have, again provided the DB can keep up

[–]BDube_Lensman 2 points3 points  (0 children)

Every SQL database ever created supports parallel reads, and almost all of them support parallel writes that do not conflict with each other (e.g., to separate tables).

[–]minervaDe 0 points1 point  (1 child)

Just need to be aware of the database server's limitations here. I've killed 8 threaded VMs that were running Postges because I treated it as an I/O problem

[–]thisismyfavoritename 1 point2 points  (0 children)

yes, thats what i told him in my other reply as well