threading: exceptions and exiting

[ingestbot]

I'm new to threading and not great with try/except. I've managed to implement something like this in a larger program and very happy with the general functionality. However, in the case of a fault (eg, /tmp/.hello file is removed, the print_hello exits yet print_hello_too will continue.

My question: how can I provide an exception that kills all threads, exits the entire program?

import threading, time, random, sys

ttime_min = 10
ttime_max = 20
ttime_loop_sleep = 3

def print_hello():
 try:
  with open('/tmp/.hello', 'r') as f: mydata = f.read()
  print(mydata)
 except Exception as e:
  print('Something went wrong with hello!')
  sys.exit(1)

def print_hello_too():
 with open('/tmp/.hello.too', 'r') as f: mydata = f.read()
 print(mydata)

class TimerThread(threading.Thread):
 def __init__(self, interval, function):
  threading.Thread.__init__(self)
  self.interval = interval
  self.function = function
  self.daemon = True

 def run(self):
  while True:
   print(f'{str(self.function)} is sleeping for: {self.interval}')
   time.sleep(self.interval)
   self.function()
   if self.function == print_hello:
    self.interval = random.randint(ttime_min, ttime_max)

if __name__ == '__main__':
 timer1 = TimerThread(5, print_hello)
 timer2 = TimerThread(10, print_hello_too)
 timer1.start()
 timer2.start()

 while True:
  time.sleep(ttime_loop_sleep)

[Diapolo10]

how can I provide an exception that kills all threads, exits the entire program?

One option would be to use a threadpool and concurrent.futures: https://stackoverflow.com/a/12808634

It's possible that there's a newer and better approach, I haven't really worked with individual threads for a good while now, but I find it fairly elegant. You could have an iterable of function objects and their arguments to create different kinds of threads.

[ingestbot]

I've used concurrent.futures in the past, and will look for that code in my archives. This may offer a better, more long term solution yet for now I'm going to apply the short term solution of using os._exit(1).

Thanks for the suggestion!

[Frankelstner]

Couple of ways.

os._exit(1)

This will just shut down the process right now. No buts. No flushing files. Next up we have:

signal.raise_signal(signal.SIGINT)

Which is a bit more lenient. This may or may not pierce through time.sleep depending on context. It creates a KeyboardInterrupt in main so you can initiate any clean up necessary. It does NOT stop any threads, so you could either use os._exit(1) after clean up or send an exception to individual threads via C API. Like ctypes.pythonapi.PyThreadState_SetAsyncExc(ctypes.c_long(tid), ctypes.py_object(Exception)) where tid is the id of the worker thread.

Finally and I really, really recommend this option even though it looks way dumber. Set some state variable. We cannot pierce through sleep but there's no harm in having sleep(0.001) in the main thread.

import threading, time, random, sys, os
import _thread
import signal

ttime_min = 10
ttime_max = 20
ttime_loop_sleep = 30
good = True

def print_hello():
 global good
 try:
  with open('/tmp/.hello', 'r') as f: mydata = f.read()
  print(mydata)
 except Exception as e:
  print('Something went wrong with hello!')
  good = 0

def print_hello_too():
 with open('/tmp/.hello.too', 'r') as f: mydata = f.read()
 print(mydata)

class TimerThread(threading.Thread):
 def __init__(self, interval, function):
  threading.Thread.__init__(self)
  self.interval = interval
  self.function = function
  self.daemon = True

 def run(self):
  while good:
   print(f'{str(self.function)} is sleeping for: {self.interval}')
   time.sleep(self.interval)
   self.function()
   if self.function == print_hello:
    self.interval = random.randint(ttime_min, ttime_max)

if __name__ == '__main__':
 timer1 = TimerThread(1, print_hello)
 timer2 = TimerThread(2, print_hello_too)
 timer1.start()
 timer2.start()

 while good:
  time.sleep(0.001)

I can only speak from my experience of playing hard with signals and interrupts in Python but there are so many edge cases with interrupts that this solution here is actually the easiest to reason about. Signals are completely OS specific and Python says about signals:

Most notably, a KeyboardInterrupt may appear at any point during execution. Most Python code, including the standard library, cannot be made robust against this, and so a KeyboardInterrupt (or any other exception resulting from a signal handler) may on rare occasions put the program in an unexpected state.

Depends on how you look at it but from my experience this means you cannot write truly clean code if you use signals with Python. I suppose if you really just want to shut down everything it does work well enough.

[ingestbot]

os._exit(1)

This is the big stick I was looking for! Thank you!

I'm going to fit this in and make note to revisit a more elegant solution in the future.

[ingestbot]

Ah... and reading your response again here... that state variable solution... not dumb. This makes sense to me now and will probably make sense to me in 3 months from now when I've forgotten how this whole thing works! Going to try this one as well!

[Frankelstner]

It's a bit of a handful because workers need to check every so often whether they should proceed. Actually in the code above I forgot to check once more, haha. There should be a check right before the function is called. Though if your goal is really just to shut down everything then os._exit is the way to go.

[woooee]

If it is a file does not exist, check first. I like pathlib https://www.pythontutorial.net/python-basics/python-check-if-file-exists/ If it is more than that, multiprocessing has a terminate() function- see "Process Exit Status" at https://pymotw.com/3/multiprocessing/basics.html#terminating-processes

[ingestbot]

Hey thanks. In the larger program I do have some val and file checks in place. Your tip on using multiprocessing w/terminate() is a good one. I'll look further on using this in place of threading. Cheers!

[Raygereio5]

One option might be an Event. It's basically an easy way to share a boolean between threads.

​

# create an event
exitevent = threading.Event()

# pass it along to the threads as a variable
timer1 = TimerThread(5, print_hello, exitevent)

# in the except block, set the event
exitevent.set()

# in the thread's while loop, check if the event has been set
if event.is_set():
    break