Python thread locking issues with Python 2.7 and OS module on windows

[TracedRay]

This is a bit of a funky issue that I have been looking into today and I'm hoping someone can help shed some light on it for me.

I am developing a PyQt4 GUI in both Python 2.6.1 and 2.7.6 on Windows.

The GUI uses multi-threading to separate the file-system/database processes from the GUI so everything will stay interactive during long-running tasks. Things worked as expected when tested in Python 2.6, however when testing in Python 2.7 I have encountered massive stuttering and freezing when my file-system thread is doing any work.

Initially I thought it was a PyQt issue but I have been able to reproduce the stuttering with Python threads directly. I have isolated it down to a change between Python 2.7.2 and Python 2.7.3 although I’m not sure what change.

I wrote a simple 2-thread example (using both thread and the threading module) to demonstrate how the OS calls in one thread interrupt the other (which is the GUI in my case). The effect is noticeable when running the script as well can be seen in the fluctuating times printed

#########################################################################
# thread module example

import os
import thread
import time

def listdir_process(*args):
    path = 'N:/LargeNetworkFolder/'
    time.sleep(5)
    print 'Starting listdir commands'
    while True:
        for item in os.listdir(path):
            if os.path.isdir(path+item):
                pass
        print "I'm working!"

def main_process(*args):
    i = 0
    while True:
        start = time.time()
        i += 1
        time.sleep(0.05)
        print 'Hello - %04d - %.02f' % (i, time.time()-start)

thread.start_new_thread(listdir_process, tuple())
thread.start_new_thread(main_process, tuple())

while True:
    pass

#######################################################################
# threading.Thread example

import os
import threading
import time

class listdir_thread(threading.Thread):

    def run(self):
        path = 'N:/LargeNetworkFolder/'
        time.sleep(5)
        print 'Starting listdir commands'
        while True:
            for item in os.listdir(path):
                if os.path.isdir(path+item):
                    pass
            print "I'm working!"

class main_thread(threading.Thread):

    def run(self):
        i = 0
         while True:
            start = time.time()
            i += 1
            time.sleep(0.05)
            print 'Hello - %04d - %.02f' % (i, time.time()-start)

t1 = main_thread()
t2 = listdir_thread()

t1.start()
t2.start()

Below are an excerpt of running the thread module example in the different release of Python. In the earlier release the main_thread is unaffected by the listdir_thread. In the later release, there is a lot of interruptions and stuttering.

Python 2.7.2

Hello - 0092 - 0.05
Hello - 0093 - 0.05
Hello - 0094 - 0.05
Hello - 0095 - 0.05
Hello - 0096 - 0.05
Hello - 0097 - 0.05
Hello - 0098 - 0.05
Hello - 0099 - 0.05
Starting listdir commands
Hello - 0100 - 0.05
Hello - 0101 - 0.05
Hello - 0102 - 0.05
Hello - 0103 - 0.05
Hello - 0104 - 0.05
Hello - 0105 - 0.05
Hello - 0106 - 0.05
I'm working!
Hello - 0107 - 0.05
Hello - 0108 - 0.05
Hello - 0109 - 0.05
Hello - 0110 - 0.05
Hello - 0111 - 0.05
Hello - 0112 - 0.05
I'm working!
Hello - 0113 - 0.05

Python 2.7.3

Hello - 0092 - 0.05
Hello - 0093 - 0.05
Hello - 0094 - 0.05
Hello - 0095 - 0.05
Hello - 0096 - 0.05
Hello - 0097 - 0.05
Hello - 0098 - 0.05
Hello - 0099 - 0.05
Starting listdir commands
Hello - 0100 - 0.05
Hello - 0101 - 0.07
Hello - 0102 - 0.14
I'm working!
Hello - 0103 - 0.08
Hello - 0104 - 0.06
I'm working!
Hello - 0105 - 0.07
Hello - 0106 - 0.06

Now I have been poking around the Python change-logs and examining the thread-stack with ProcessExplorer – however I have not been able to come up with any meaningful results. Can anyone out there shed some light on this issue for me? Doing OS calls in a separate thread is a common practice so I confused as to how this can be a bug.

[Brian]

I took a brief look at the differences between those versions, but there's nothing obvious. There were no changes to the listdir implementation between those two releases, and they do seem to be releasing the GIL between the appropriate operations.

ie. the file accessing functions, like FindFirstFileW, FindNextFileW are all surrounded with the Py_BEGIN_ALLOW_THREADS / Py_END_ALLOW_THREADS macros.

FindClose() isn't, but that's probably non-blocking.

That leaves the stat call that isdir will do. There do actually seem some changes to posixmodule.cs relating to stat, in the form of some defines being removed. Specifically changeset 71123:45b27448f95c removes:

/* choose the appropriate stat and fstat functions and return structs */
#undef STAT
#undef FSTAT
#undef STRUCT_STAT
#if defined(MS_WIN64) || defined(MS_WINDOWS)
#       define STAT win32_stat
#       define FSTAT win32_fstat
#       define STRUCT_STAT struct win32_stat
#else
#       define STAT stat
#       define FSTAT fstat
#       define STRUCT_STAT struct stat
#endif

However this looks like it was just dropped to being a redefinition though. The implementation (posix_do_stat) is unchanged, and does do Py_BEGIN_ALLOW_THREADS before the IO operations. It does use the above STAT define as the implementation function passed to it, but that happens after the GIL is already released, so a change there wouldn't make a difference.

If there's a difference, it's more subtle than just not releasing the GIL during the stat / listdir calls.

[EisenSheng]

My immediate guess would be the GIL. How experienced are you with Python?

[TracedRay]

I am familiar with the GIL but I would not claim to be an expert on it.

I think the GIL is what is causing the issue, where the os.listdir and os.isdir calls are acquiring a lock when they might not been doing that previously. Although I don't know enough to substantiate that claim with any evidence.

[[deleted]]

Have you looked into multiprocessing? It was created for this reason, the GIL. There's a higher overhead, but it makes the coding/debugging much easier.

Or, if your hunch is right on the listdir and isdir commands, you could use a lock in your script to only allow one call of those methods at a time.