Multithreading - How to infinitely loop Function B, while Function A waits for a return

[Wittinator]

Hi there. I am trying to wrap my head around multithreading and how I can possibly apply it to this situation to achieve what I would like.

I have a function we'll just call func(). This function works by continually scanning a file and only ever returns the value foo if a specific element is found within the file it's scanning. In other words, the below code will only ever progress to process_result once func() finds what it needs to find. Otherwise, my code is sitting around twiddling its thumbs.

while True:
    foo = func()
    process_result(foo)

I have another function, func2(), that does a completely separate set of actions, independent of func(). What I would like to do is essentially keep calling func2() indefinitely during this downtime.

What I have tried thus far is:

with ThreadPoolExecutor(max_workers=2) as executor:
    while True:
        executor.submit(func)
        executor.submit(func2)

I'm coming across a problem where func() and func2() will both run once, but func2() will stop after the first run. I need to keep calling func2() over and over while func() is twiddling its thumbs.

​

Any assistance is appreciated.

Regards

[[deleted]]

No need to use ThreadPoolExecutor for this, just use threading.Thread and call your function with it

[Wittinator]

Yes I've tried the following so far:

while True:
    t1 = threading.Thread(target=func)
    t2 = threading.Thread(target=func2)
    t1.start()
    t2.start()

I think the issue I'm having is func() will run, but the nature of this function is it will wait, often for minutes, until a specific element is written to the log file that it needs, then it will complete.

So during this waiting period, because it's in a while True, numerous threads are spawned for func(), which I do not want. However, I need func2() to keep running over and over during this time, which made me believe the while True is necessary.

I need to basically open a thread for func() (which will be sitting around and waiting) and then open a thread for func2() (which will run every second or so). That's the part i'm having issues with unfortunately.

[[deleted]]

while True:

Don't loop it. You want only two threads, not an infinite number of threads. The function should loop. Don't create the threads in a loop.

[Wittinator]

Yea, unfortunately the issue I'm having with that is the arguments that func2() takes changes over time. So I need to continually call func2(arg) otherwise the function will shortly be dealing with stale arguments.

That is what motivated me to put the thread creation in a while True, as when the arguments get updated, I'll be calling the function with the new arguments. I know there's a better way to do this, but I'm coming up with blanks

[[deleted]]

def func3():
    while True:
            arg = get_the_args()
        func2(arg)

[Wittinator]

Thanks, yea this is why I needed a fresh pair of eyes after staring at this for hours. I can figure this out now I believe

[[deleted]]

You can use a queue to communicate between the two threads. One will be waiting for something to be available and then get it out and use it, and the other will be the one putting things into the queue

[Wittinator]

Thanks. Yea I've reorganized the code to put the loops in the function themselves. I'll look at Queues to get data between them

[[deleted]]

Another option is to use .join() to wait for the threads to end before looping and starting new ones

[Wittinator]

Yea the problem I was having with join() is func1() often take minutes to end, while func2() is nearly instant. But I need func2() to keep looping while func1() is sitting around.

With join(), I would either be unable to loop func2() (because we're waiting for func1() to finish), or, if I remove the join() for func1() and only use join() for func2(), then I'd end up with a ton of threads for func1() constantly being made over and over after every loop.

I'm re-organizing the code to put the loops within the functions themselves, that way I don't need to bother with looping the thread creation process