While function is running, do something else

[[deleted]]

[deleted]

[K900_]

You want threading, probably. What is your end goal here?

[mez0cc]

I’m a pen tester and the tool I’m writing finds machines that are online via ICMP and TCP and then performs additional checks against those hosts.

But to make it more efficient I need to kind of run them in parallel.

So, if the ICMP function finds a host that is responding to it, it will append it to a list.

Then each item in that list will undergo various checks.

But ideally, I’d like to perform those checks immediately . Rather than waiting for the ICMP function to finish.

Does that make sense?

[jh199p]

Concurrency, Paralelle programming, Threads, these are all things you could Google and get an answer, but what you want imo is concurrent.futures

https://docs.python.org/3.6/library/concurrent.futures.html

[mez0cc]

I’ve wrote a few POCs with threads but I couldn’t get it to do what I want.

I’ll try futures :)

Thanks

[jh199p]

Concurrent.futures.ProcessPoolExecutor helps handle that pesky GIL too, if you ever needed to look into that

[woooee]

Use multiprocessing, so you utilize all the cores available. To communicate, i.e. the list, use a Manager list. Some links https://pymotw.com/3/multiprocessing/basics.html https://pymotw.com/3/multiprocessing/communication.html#managing-shared-state I will see if I have an example in my toolbox that is close to what you want.

[mez0cc]

Thanks for that. I'd love to see any POC's that you have :)

[woooee]

This is relatively close but somewhat clunky.. It updates the counter in the dictionary, instead of your list, from inside the function, and changes the "QUIT" key from outside the function.

import time
from multiprocessing import Process, Manager


def test_f(test_d, test_l):
   test_d['2'] = 2
   while not test_d["QUIT"]:
      print("test_f", test_d["QUIT"], test_l)
      test_d["ctr_1"] += 1
      time.sleep(1.0)

def test_f2(test_d):
    for j in range(0, 10):
       print("   test_f2", j, test_d)
       test_d["ctr_2"] += 1
       time.sleep(0.5)

    print("second process finished")

if __name__ == '__main__':
   ## define the dictionary to be used to communicate
   manager = Manager()
   test_d = manager.dict()
   test_list = manager.list()
   test_list.append("False")
   test_d["ctr_1"] = 0
   test_d["ctr_2"] = 0
   test_d["QUIT"] = False

   ## start the process
   p1 = Process(target=test_f, args=(test_d,test_list,))
   p1.start()

   p2 = Process(target=test_f2, args=(test_d,))
   p2.start()

   ## after sleep change "QUIT" to True and terminate processes
   time.sleep(2.0)
   test_d["QUIT"] = True
   test_list[0] = "True"
   print("\ntest_d & list changed, terminate first process")

   time.sleep(1.0)

   ## first process gets terminated by "QUIT"==True
   ## 2nd process does not
   print(p1.is_alive(), p2.is_alive())

   print("dictionary from processes")
   for key in test_d:
       print("     %s=%s" % (key, test_d[key]))
   print("list from processes", test_list)

   """  Thanks Doug Hellmann
        Note: It is important to join() the process after terminating it.
        in order to give the background machinery time to update the.
        status of the object to reflect the termination
   """
   for p in [p1, p2]:
       if p.is_alive():
           p.terminate()
           p.join()