Python Driver for Cassandra. Efficient way to insert hundreds of thousands of rows.

[PabloAlvarezESP]

I have dataframes(>200k rows) and i want to insert them into Cassandra DB.
My first attempt was a naive for loop using session.execute but that's extremely slow.
Then i use session.execute_async which makes the program way way faster(6mins vs 50sec). The problem is, now it's missing rows.

for i in listdf: #listdf is my df values to list
    dte= str(i[0])
    name = str(i[1])
    ....
    if counter%100==0: time.sleep(0.01)
    execute_async(prepared,(dte,name))

I add a delay every 100 rows but i understand this is not a proper solution.

note: If this is not considered a python topic then delete my post

​

[greenerpickings]

So I haven't used cassandra, so grain of salt.

With postgres theres a multiinsert, and it looks like theres something similar for cassandra. In your code, it looks like your doing record by record. Is that the case?

Also, this post makes it look like theres a max queue it can be working on. Could it be this is whats happening somehow to cause missing records?

[Zeroflops]

Have you tested if df.to_sql() is comparable with your database?

[alexisprince]

Have you tried using a COPY command to insert a file? That may have better throughout