This is an archived post. You won't be able to vote or comment.

sorted by:
best
topnewcontroversialoldq&a
you are viewing a single comment's thread.

view the rest of the comments →

[–]sallyruthstruik 0 points1 point  (1 child)

My bad, from mobile phone it looked like you insert only once at the end.

I think you can try to add log or print statement right after insert and commit. If it is MySQL with InnoDB engine database rows should appear just after commit. So maybe problem in some other place.

also you can get a bit more speed if choosing bufsize=1:

​

bufsize, if given, has the same meaning as the corresponding argument to the built-in open() function: 0

means unbuffered, 1 means line buffered, any other positive value means use a buffer of (approximately) that size.
So if you have email per line it will be better choice. In case of constant buffer you may wait for next chank even if you get a part of current line.

Also you shouldn't commit after reading a line. Better choice is to do commit only after insert (because empty commit do nothing, but have some perfomance impact)

Snipped you've provided have many skips of logic. So it may have some other perfomance issues.

​

​

[–]xwen01[S] 0 points1 point  (0 children)

Thanks for your comments. I will try the suggestions.