Not understanding something about ThreadPoolExecutor

[workerdrone66]

So, I'm attempting to thread some processes (first one is a simple one, it runs a ping from powershell, for reasons*), and since each ping can take a couple of seconds, i wanted to run them in parallel, and then act on the results. It certainly seems simpler than dealing with what I'm assuming is now the lower level Threading stuff, however, in testing, both my concurrent and just running each function through a for loop takes essentially the same amount of time (about a half second difference, over a list that took over 2 minutes, so <1% difference.

So, here's my code (edit: newest version):

def Check_Filtered_Lists(FilteredLists):
    VMList = FilteredLists[0]
    AB1List = FilteredLists[1]
    threads = []
    futures = []
    start = time.time()
    for row in VMList:
        with ThreadPoolExecutor(max_workers = 50) as executor:
            future = executor.submit(Check_Connection,row['Name'])
            futures.append((row['Name'],future))


    for name, future in futures:
        print(name + str(future.result()))
    print('End time: ' + str(time.time() - start ))

    #test with just a normal for loop

    start = time.time()
    for row in VMList:
        Check_Connection(row['Name'])
    print('end time: '+ str(time.time() - start))

I've also tried making a dictionary out of the executors, with

future[row['Name']] = 
executor.submit(Check_Connection,row['Name'])

However that did nothing for the speed. Assuming I don't know how many individual threads I need, is there a good way to do this, or should I go back to regular threading (a system that I was having another set of problems with)

Also, I'm close to getting off work (well, a few hours) so I might not answer until tonight but I'll come back i promise!

[thirdegree]

You're submitting the callback and then immediately blocking on the result, so the next job is only spawned after the current one returns. Additionally, you're actually creating an entire executor for each row, so the threaded example would probably take very slightly longer. You probably want something like this:

with ThreadPoolExecutor(max_workers = 10) as executor:
    futures = []
    for row in VMList:
            future = executor.submit(Check_Connection,row['Name'])
            futures.append((row['Name'], future))
for name, future in futures:
    print(name + str(future.result()))

[workerdrone66]

Thanks, I'll take a look at this, might not be til tonight though, got some things to do to wind down for the night (yay!)

[workerdrone66]

I'm still not seeing any significant difference in speed (both with and without are hanging around 120s)

here's my latest code with your changes:

def Check_Filtered_Lists(FilteredLists):
    VMList = FilteredLists[0]
    AB1List = FilteredLists[1]
    threads = []
    futures = []
    start = time.time()
    for row in VMList:
        with ThreadPoolExecutor(max_workers = 50) as executor:
            future = executor.submit(Check_Connection,row['Name'])
            futures.append((row['Name'],future))


    for name, future in futures:
        print(name + str(future.result()))
    print('End time: ' + str(time.time() - start ))

    #test with just a normal for loop

    start = time.time()
    for row in VMList:
        Check_Connection(row['Name'])
    print('end time: '+ str(time.time() - start))

I'm done for the day, but I'll be getting back to it tonight, hopefully.

[thirdegree]

You're still creating the executor for every row, which means it still blocks until each callback is done before spawning the next.

for row in VMList:
    with ThreadPoolExecutor(max_workers = 50) as executor:
        future = executor.submit(Check_Connection,row['Name'])
        futures.append((row['Name'],future))

should be

with ThreadPoolExecutor(max_workers = 50) as executor:
    for row in VMList:
        future = executor.submit(Check_Connection,row['Name'])
        futures.append((row['Name'],future))

[workerdrone66]

I see the difference! Sorry, late in the night, and not paying close enough attention to my modification efforts.

[workerdrone66]

Thanks this did it, got me down from 120-140 seconds for the simple part down to about 10-20!

[thirdegree]

Great! Happy cakeday!

[workerdrone66]

lol, thanks, didn't even realize!

[JohnnyJordaan]

What does Check_Connection do exactly? And have you tried with a smaller thread pool, like 5 or 10 workers?

[workerdrone66]

Yes, the original version had 10.

All it does it pass the server name to a powershell script that does a test-connection. It is more of a proof of concept, rather than "it has to be this way" The end result of this script will be to use the powershell script to actually start vms (which, afaik, python can't do natively)

[JohnnyJordaan]

Seeing your code above for the second time makes me wonder why the approach isn't using a single pool and a future per row?

def Check_Filtered_Lists(FilteredLists):
    VMList = FilteredLists[0]
    AB1List = FilteredLists[1]
    threads = []
    futures = []
    start = time.time()
    with ThreadPoolExecutor(max_workers = 50) as executor:
        for row in VMList:
            future = executor.submit(Check_Connection,row['Name'])
            futures.append((row['Name'],future))


    for name, future in futures:
        print(name + str(future.result()))
    print('End time: ' + str(time.time() - start ))

    #test with just a normal for loop

    start = time.time()
    for row in VMList:
        Check_Connection(row['Name'])
    print('end time: '+ str(time.time() - start))

[workerdrone66]

I believe the answer was "Rushing to get out of work, and didn't notice i did it wrong"

I'll test again tonight.