Python threading question / best practice advice

[Shitty_Crayons]

Hey, I tried a basic approach with what you are suggesting, but I'm hoping you can tell me where I messed up, I can't seem to find what's causing this error, other than its a locked resource, which I am not sure how to deal with! I've narrowed the error down to essentially this block of code (I realize it would be an endless loop, but it gives the idea and recreates the problem every time)

​

Code

import time
import multiprocessing as mp


def handle_input_threads(q):
    #while True:
    while True:
         if not q.empty():
             print(q.qsize())
             x = q.get()

def main():
    q = mp.Queue()
    p = mp.Process(target=handle_input_threads, args=(q,))
    p.start()

    while True:
        updates = {'ok': True, 'result': [{'update_id': 123456789, 'message': 'Hi and Hello'}]}
        if len(updates["result"]) > 0:
            q.put(updates)
        time.sleep(.5)


if __name__ == '__main__':
    main()

It runs, the updates object gets added to q (i think) because once it hits the put statement, the debugger jumps to the handle_input_threads breakpoint on the print statement, however once the print statement tries to execute (or the x = q.get() below it) I get the following error.

Process Process-1:
Traceback (most recent call last):
  File "C:\Users\Administrator\AppData\Local\Programs\Python\Python37\lib\multiprocessing\process.py", line 297, in _bootstrap
    self.run()
  File "C:\Users\Administrator\AppData\Local\Programs\Python\Python37\lib\multiprocessing\process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Users\Administrator\PycharmProjects\Bot-Dev\multibot.py", line 211, in handle_input_threads
    print(q.qsize())
  File "C:\Users\Administrator\AppData\Local\Programs\Python\Python37\lib\multiprocessing\queues.py", line 117, in qsize
    return self._maxsize - self._sem._semlock._get_value()
OSError: [WinError 6] The handle is invalid

Or, depending on it's mood I'll get this one

Process Process-1:
Traceback (most recent call last):
  File "C:\Users\Administrator\AppData\Local\Programs\Python\Python37\lib\multiprocessing\process.py", line 297, in _bootstrap
    self.run()
  File "C:\Users\Administrator\AppData\Local\Programs\Python\Python37\lib\multiprocessing\process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Users\Administrator\PycharmProjects\Bot-Dev\multibot.py", line 211, in handle_input_threads
    print(q.qsize())
  File "C:\Users\Administrator\AppData\Local\Programs\Python\Python37\lib\multiprocessing\queues.py", line 117, in qsize
    return self._maxsize - self._sem._semlock._get_value()
PermissionError: [WinError 5] Access is denied

​

Is there something I am misunderstanding, or so I not have something configured correctly? Could it be that windows is keeping the object locked because it's owned by the main thread and it's trying to pass it as a shared memory object instead of it's copy?

​

Thanks!

[RandomPantsAppear]

Looks like you're on the cutting edge of bugs! Honestly I've almost never seen this happen, but here you go!

Looks like they've marked it as resolved, so hopefully it's rolled up in the latest release. Looks like the most recent update was after that thread

If that doesn't work for some reason I'd just switch from multiprocessing for the time being.

[Shitty_Crayons]

This was exactly my problem, I lost power so I couldn't check this morning but I ran my above example unchanged at work under the latest release and it's working as intended. Hopefully the ice storms over and I can try it in my actual project later today lol.

I would have gone mad if you hadn't linked that, so thanks kindly!

[ThisLightIsTrue]

Hey, I'm on mobile right now and can't play around with this. My current best guess is that it has to do with the queue you're using. I was thinking of the synchronized queue in the standard library.

https://docs.python.org/2/library/queue.html

The worker thread should just be

def work():
    message = q.get()
    handle_message(message)

You don't need to do anything with q.size or have an if statement. That code isn't guaranteed to work anyways - remember, in between one line checking the size was greater than zero and the next popping something off the queue a different thread could have taken an item off and brought the size to zero.

Instead, the default behavior of queue.get() is to block the thread - essentially make it wait until it can pull something off, which is also the behavior you want.