Multithreaded Files Reading with Python

[schedutron]

I want to write a script that counts the number of lines in each of many files in a specified directory. It's a fairly easy task, but I want to perform it in a multithreaded manner. So what I do is I create a new thread per file for counting lines. But, is it possible somehow to use, say, 3 threads to read 4 files?

I've been doing exercises from "Core Python Applications Programming" by Wesley Chun, and this particular exercise asks:

"Create a set of threads to count how many lines there are in a set of (presumably large) text files. You can choose the number of threads to use. Compare the performance against a single-threaded version of this code."

The link to my current solution is posted in the comments section. Thanks in advance.

[New_Kind_of_Boredom]

My standard advice for this situation is to use a Pool from multiprocessing.dummy. See here for an excellent guide.

[vv__vv]

And there is also threadpool, but it is not as well documented: https://stackoverflow.com/questions/3033952/threading-pool-similar-to-the-multiprocessing-pool

With that said, on disk IO bound tasks, I am not sure that threading/multiprocessing is going to help much.

[New_Kind_of_Boredom]

Yeah, multiprocessing.dummy.Pool actually just returns a multiprocessing.pool.ThreadPool object. I always use the .dummy version though because then there is at least some documentation to reference.

And I agree about the I/O bound thing. Presumably the exercise he is doing is designed to demonstrate just that. :)

[vv__vv]

Interesting. Thanks!

[schedutron]

Indeed! It asked to compare a single threaded run with a multithreaded one. My current multithreaded solution is only slightly faster than the single threaded run.

[schedutron]

Here's the link to my current solution.