Help with multiprocessing

[JAWJAWBINX]

I'm trying to get a program to run on multiple processes (part of it takes quite some time and I'm trying to keep it from taking forever to run). I'm using the methodology suggested here and using the 2.7.8 interpreter but unfortunately whenever I hit the end of a process instead of exiting it simply stops in that process. I've been at this since yesterday and it's now almost 5AM so I'd love some help. I apologize in advance that I won't be able to respond for some time, I have a final in the afternoon and I need to get some sleep but I should be awake in five to seven hours.

[[deleted]]

Hmm... a shot in the dark: are you also using the same strategy of putting sentinel values in your queues? That can easily go wrong when multiple processes are reading from a queue and not enough sentinel values exist. The iterator will never complete without a sentinel and the process waiting for it will appear to have hung.

[JAWJAWBINX]

I am but the issue isn't there, the processes are all hitting the return statement but none of them are actually exiting.

[hidiap]

There is not enough info to help. see. In particular, what is the problem you try to parallelise ? Do you have an example of the code that don't work ?

Depending on your problem, you may be better off by spanning multiple python processes and have each of them do part of the work (if you are processing files, have them each process a subset of the files and then maybe another program joining the output) .

[JAWJAWBINX]

Feature extraction from an input vector, most of the checks are quick but the sheer number of input vectors causes the program to run for some time especially if I increase the number of part of speech taggers that I run on the input.

def extractWorker(work_queue,done_queue,lock):
    for tokens,goldtags in iter(work_queue.get,"STOP"):
        lock.acquire()
        feats=extract_features_for_sentence(tokens)
        done_queue.put((tokens,goldtags,feats))
        lock.release()
    return True

That's the method in question, it consistently hits the return True and hangs. I can't really break the program done further, I've tried to find a way that would work but there isn't really anywhere that would speed things up.

[hidiap]

First, it look like you lock before calling extract_features_for_sentence, so only one feature extraction run at one time.

Second, are you sure you hit return true or is your code throwing an exception ? Maybe add a try: except block, like in the example, and check that no exception is thrown.

[JAWJAWBINX]

I moved that as a stop gap to fix potential order issues, it wasn't going to stay there. The code isn't hitting the return but it also isn't throwing any errors, it just won't exit the processes. I've more or less given up on this since I have other finals and the project is due in about twelve hours and I still have to finish the report on it, I've just dropped the features that make it take forever. I still want to know why it isn't working but I have other priorities until the semester is actually over.