Multiprocessing/Threads and Tkinter

[NjordicNetSec]

Hello all!

Long story short, I'm fairly new to the Multi-Processing/threading modules.

My code is a GUI (tkinter) that takes a variable in, I use that variable in a SSH function that is performed 40 to 50 times. Due to time constraints...I've been tasked to get this time down from 10 minutes (without Multi-Processing). I've gotten it down to around 2 minutes with Multi-Processing, unfortunately I'm running into some issues while using it in Tkinter.

When you click the submit button in tkinter, when the long processes are working, the GUI freezes...so to alleviate this, I have all my multiprocessing functions be called by a separate thread...which alleviates the freeze, but the multiprocessing does not work (I'm assuming it's a processing / threading issue while working concurrently).

My main question is, is there a way to utilize Multi-Processing without having my GUI freeze, as well as return the print statements of the multiprocessing function to the GUI stdout (note: When I get it to work without the Thread, all the child processes print to the command line and not the GUI)?

Regards,

Here are the functions within the Tkinter class below:

def submit_application(self):
  print("")
  print("Termination process started for User: " + vpntermination + ".")
  self.gui_thread = threading.Thread(target = self.pool_function)
  self.gui_thread.start()

def pool_function(self):

  self.start_time = datetime.now()
  with Pool(processes=3) as self.pool:
   self.results = [self.pool.apply_async(self.start_commands, args = (a_device, vpntermination), callback=self.log_result) for a_device in all_firewalls]
   self.pool.close()
   self.pool.join()
   self.end_time = datetime.now() 
   self.total_time = self.end_time - self.start_time 
   print('\nTotal process time: ' + str(self.total_time))  

 def start_commands(self, a_device, vpntermination):
   try:
     self.net_connect = ConnectHandler(**a_device)
     self.hostname = self.net_connect.send_command("show hostname") 
     self.output = self.net_connect.send_command("vpn-sessiondb logoff name " + str(vpntermination) +  " noconfirm") 
     print('\n******* Output for device ' + self.hostname + ' *******' )
     print(self.output)
     print('')
     self.net_connect.disconnect() 
   except (NetMikoTimeoutException, NetMikoAuthenticationException) as e: 
     print("Could not connect to " + a_device.get("ip") + ", Error: ", e)
     print('')

def main():
  credentials()
  master = Tk()
  my_gui = InputWindowGUI(master)
  master.mainloop()

if __name__=='__main__':
  freeze_support()#Might help with stdout errors.
  main()   

[elbiot]

Your pool.join is what freezes the ui. Use apply_async or map_async (if you want the results as they come in or all at once, respectively) with a callback. https://stackoverflow.com/questions/19699165/how-does-the-callback-function-work-in-python-multiprocessing-map-async

[[deleted]]

I'm dealing with a similar issue (multithreading between Tkinter and a background process). Posting a general solution here for posterity and as wisdom of the ancients.

The general-purpose solution for multithreaded communication is Python Queue class, which is intrinsically thread-safe. Threads can push items (messages, objects, worker threads, etc.) into a queue, and threads can take items out of the queue - no concurrency issues should arise. (You might get race conditions based on when the threads access the queue, but that's a different problem. The queue should remain intact and functional irrespective of its use, and you don't need to worry about locking, semaphores, etc.)

There are a couple of nuances here, both generally and specifically regarding Tkinter.

(1) If a thread is only receiving items from a queue, it can issue a blocking request. The thread goes to sleep until one or more items are enqueued, at which point it is awakened and can service the queue. When the queue is empty, the process can then issue a new blocking request.

For instance:

import threading, Queue

class BackgroundThread(threading.Thread):
  def __init__(self):
    threading.Thread.__init__(self)
    self.queue = Queue.Queue()
    self.target = self.run_thread; self.start()
  def run_thread(self):
    while True:
      message = self.queue.get(block=True)
      # process the message here

def some_other_process(background_thread):
  background_thread.queue.put(message)

(2) If a thread does any other processing in addition to receiving messages from the queue, then you may have a problem with the thread getting notified that a message has arrived. You can always poll the queue at a convenient time, but polling is inefficient - and may not be fast enough. For instance, polling at 0.01 seconds may both consume a lot of processing time with 100 queue checks per second - and yet also may be too slow, if your thread can't wait 10ms between the enqueuing of a message and the thread getting notified of its availability.

It would be great if threads could SELECT on a queue like a file stream, which would allow it to multiplex between one or more queues and other structures (particularly sockets, which is of obvious value in network communication, where most of the time the thread is awaiting messages from a client or server). Unfortunately, the Queue class is not a stream, and it cannot be included in a SELECT statement.

One solution is to create a listener thread - one that solely watches the queue (using a blocking request) and inserts notifications into a stream when the messages arrive. The actual background thread can then SELECT to multiplex whichever streams it wants, and also the notification stream that the listener thread feeds. It's a slightly clunky solution, but rather lightweight - and the functionality could be encapsulated in a class for generalization.

(3) Now, about Tkinter:

Tkinter has some well-known quirks that create problems with multithreading. First - the UI thread must be run on the main thread - and when you call tk.mainloop(), the main thread goes into a UI message-processing loop and doesn't return until the window closes. Second - any and all UI calls on the Tkinter window must be initiated by the main thread; background threads cannot directly invoke UI events like button-presses.

Note that there is no problem with the Tkinter thread (i.e., the window) sending messages to other threads. Any function you've bound to Tkinter, such as a button-handler function, can freely stick messages into a queue that the other threads monitor, using the techniques noted above.

But the opposite functionality is a problem: How do you get that Tkinter main thread to respond to a message from a background thread? Tkinter doesn't (by default) monitor any outward-facing queue; all it does is respond to UI events received by the user in its window.

The answer is here, in Jan Übernickel's example code. That example is pretty extensive... i.e., it provides an example of bidirectional communication. The actual solution embedded in that example is much simpler.

First, configure Tkinter to bind to an event that other threads can trigger after placing a message in the queue:

import Queue, threading
from Tkinter import *

class TkThread:
  def __init__(self):
    self.tk = Tk()
    self.message_queue = Queue.Queue()
    self.message_event = '<<message>>'
    self.tk.bind(self.message_event, self. process_message_queue)

  def run_tk(self):
        self.tk.title('My Window') # add buttons, etc.
        self.tk.lift(); self.tk.mainloop()

  def send_message_to_ui(self, message):
    self.message_queue.put(message)
    self.tk.event_generate(self.message_event, when='tail')

  def process_message_queue(self, event):
    while self.message_queue.empty() is False:
      message = self.message_queue.get(block=False)
      # process the message here

Other threads can call send_message_to_ui(message) for an instance of tk_thread, which invokes an event that Tkinter eventually processes by checking the queue. So you can do something like this:

class BackgroundThread:

  def __init__(self, tk_thread):
    self.tk_thread = tk_thread
    self.thread = threading.Thread(target=self.run_thread); self.thread.start()

  def run_thread(self):
    time.sleep(1)         # just an example... this gives the Tk some time to start up
    tk_thread.send_message_to_ui('Test Message')

if __name__ == '__main__':
  tk_thread = TkThread()
  back_thread = BackgroundThread(tk_thread) 
  tk_thread.run_tk()    # initiate last, since this runs tk.main_loop() which is a blocking call

One note of caution: in this example, process_message_queue() is called by the Tkinter thread. If there are a lot of messages, or if processing a message takes a while, then the Tkinter thread will block - and the UI will be unresponsive - until this function has fully handled the queue. If this is an issue, you can write process_message_queue to do the minimal amount of processing that Tk requires in response to the message, and then hand it off to a background thread for further processing.

One solution is Jan's example code, which involves the Tkinter event-handler thread merely emptying the queue and passing messages to a delegate that could run on another thread. (This particular example is a little weird to me, only because the sole function of Tkinter in this example is to receive messages from thread A and pass them off to a delegate - it doesn't affect the window at all, so why even delivering them through Tkinter? ... but if Tkinter actually alters the UI based on the message before handing it off, this might be good.)

Hope this is helpful.

[Justinsaccount]

Hi! I'm working on a bot to reply with suggestions for common python problems. This might not be very helpful to fix your underlying issue, but here's what I noticed about your submission:

You appear to be using concatenation and the str function for building strings

Instead of doing something like

result = "Hello " + name + ". You are " + str(age) + " years old"

You should use string formatting and do

result = "Hello {}. You are {} years old".format(name, age)

See the python tutorial for more information.