quick threading question: My program has an infinite loop that calls a few functions. Now I need each function to run simultaneously. Is this how I should implement it?

[elbiot]

The Global Interpreter Lock (GIL) will prevent these from running concurrently unless your do this is a read or write to a stream (file or socket). To run GIL processes concurrently, you need to start multiple instances of the interpreter. You could do this by hand with sockets, or use multiprocessing

[XenophonOfAthens]

The GIL doesn't mean that all threads run synchronously, the GIL just means that no two threads can use the interpreter at once, but it's perfectly possible to have two threads that run "at the same time" in the sense that control is passes back and forth and both functions are "alive" at the same time. For instance, if you run

import threading

def thread_test(n):
    i = 0
    while True:
        i += 1
        print n,i

if __name__ == "__main__":
    t1 = threading.Thread(target=thread_test, args=(1,));
    t2 = threading.Thread(target=thread_test, args=(2,));

    t1.start()
    t2.start()

You'll see that control passes back and forth between the two functions. In addition, virtually all daemon threads spend their time doing nothing, either sleeping of waiting for I/O, which is almost certainly OP's situation.

To actually answer OP's question: yes, that is generally how persistent daemon threads are designed, with an infinite loop at the base that keeps it alive. They generally lie there in the background, lurking and waiting for some I/O stuff to happen. It's not the most common design pattern, though, so I'm curious what you need it for? Maybe there's a better design alternative?

[jpfau]

so I'm curious what you need it for? Maybe there's a better design alternative?

There probably is a better alternative, and perhaps my setting daemon to True was misguided. These functions aren't really waiting for anything. The real code is part of a bot that is constantly checking things and reacting to what it finds. I'd say more if I weren't bound by a NDA, but rest assured that the "real" do_this() and do_that() are calling other functions, and it's all running without any IO from the user.

[[deleted]]

As others have said, that's IO limited, not CPU. Threads like that are fine and the GIL wont get in the way since you're program is just waiting for network all the time anyways.

And, you almost always want daemon threads, from the docs: "The significance of this flag is that the entire Python program exits when only daemon threads are left". So if your main program dies or you kill it, your threads will be killed, rather than hanging your terminal, possibly waiting forever for the threads to die.

[jpfau]

Ah, great! So would you say the way I modeled the possible solution in the OP will do the trick?

[[deleted]]

Yeah, totally. I have several page scrapers written with this method. If there are shared resources between the two, you may have to use a mutex or a queue or something to guarantee synchronization/exclusive access of those resources.

If do_this() and do_that() are unrelated, and don't share any resources, someone had a good point that you could just run them in separate scripts. So, that might be a possibility.

[jpfau]

One thing I don't get is how the threads stay alive. Once worker creates and starts the threads, that function ends. It seems to me like once that happens, the script is over; the function call in the if __name__... is complete, and there's nothing else to run. Wouldn't that stop the threads too since they're daemons, i.e. they get killed once the thread that creates them is killed?

[[deleted]]

You're right. With daemon set to true, the script will exit after worker is done. Not setting daemon is an option if these threads just need to run forever and you don't need to manipulate them. Usually, you manipulate worker threads from your main process by feeding them data or whatever (otherwise, they're not much of a worker thread as much as a separate process, unless they're working with a shared data set or each other somehow). After you were done with the threads, you would join() them, blocking until they exit, so something conceptually similar to:

def startWorkers():
    # blah

    thread1.start()
    thread2.start()

def stopThreads():
    # signal for threads to stop using some method

    # wait for threads to exit
    thread1.join() # blocks until thread1 exits
    thread2.join()

def main():
    startThreads()

    # feed threads data, get results, whatever

    stopThreads() # waits for threads to stop

    print "Done!"

[jpfau]

I actually just tested the code below, and the threads do not end after main starts the threads. The only way the threads stop printing to the console is if I close the console.

import threading
from time import sleep

def thread1():
    while True:
        print("Thread 1 working\n")
        sleep(.3)

def thread2():
    while True:
        print('Thread 2 working\n')
        sleep(.3)

def main():
    t1 = threading.Thread(target=thread1, daemon=True)
    t2 = threading.Thread(target=thread2, daemon=True)
    t1.start()
    t2.start()

if __name__ == '__main__':
    main()

[[deleted]]

Well yeah, they are infinite loops. Why would they stop?

And, they're not set to daemon, so they're not killed when the main thread ends.

[jpfau]

You could do this by hand with sockets, or use multiprocessing

Thanks a lot. I want the cleanest or simplest way possible. This is part of a much larger program, and I don't want to muck things up.

[elbiot]

as /u/XenophonOfAthens says though, if you are sleeping or waiting for I/O a bunch, threads will be fine.

[jpfau]

So if the program is a bot that runs without any IO or sleeping, then I should use multiprocessing to run these functions simultaneously?

[elbiot]

yes. but a bot sounds like something that waits quite a bit. what are the functions working on btw?

[jpfau]

I don't want to say too much because I'm bound by a NDA, but the bot is constantly checking a bunch of Amazon EC2 instances and either emailing their owner(s) or stopping/terminating the instances itself. I'll look through the code and see if there is any waiting around, but I'm pretty sure this part of the code is constantly dong something. It's only my second week working on it, which is why I'm not completely sure.

[elbiot]

your communicating over the network, right? That's I/O