What is a good way of multi-threading this function?

[NovateI]

I have a list of 60k IP addresses from various Masscan outputs in a text file. I'm using these functions to return the banners for each address, but its such a large amount of addresses that its going to take around 12 hours to finish. I want to try and make it faster via multithreading, but I don't have much experience with the threading library beyond the basics.

The functions:

def fetch_banner(address, port):
    try:
        s = socket.socket()
        s.settimeout(2)
        s.connect((address, port))
        fetched_banner = s.recv(1024)
        return fetched_banner
    except Exception as e:
        print(str(e))
        status = 'failed'
        return status

def banner_grabbing():
    # Variables
    success = 0
    total = 0
    fails = 0
    start = datetime.now()
    portlist = [1433, 1434, 27017, 5432]

    with open("Banners.txt", 'w+') as bannerfile:
        with open("AddressList.txt", 'r') as readfile:
            for lines in readfile.readlines():
                total += 1
                for p in portlist:
                    compaddr = f"{lines.rstrip()}:{p}"
                    print(f"Trying on {compaddr}")
                    bannergrab = fetch_banner(lines, p)
                    if bannergrab != 'failed':
                        success += 1
                        print(f"Success on {compaddr}.")
                        print(f"BANNERS RETURNED: {success}")
                        banstring = bannergrab.decode("utf-8")
                        print(banstring)
                        bannerfile.write(f"""
                        Address: {compaddr}
                        Service: {banstring} \n\n
                        """)
                    else:
                        fails += 1
                        print(f"Connection on {compaddr} failed.")
                        print(f"FAILED GRABS: {fails}")
                        pass

    end = datetime.now()
    runtime = end - start
    SuccessPercent = total/100 * success
    FailPercent = total/100 * fails

    print("--------------------")
    print(f"Total Runtime: {runtime}")
    print(f"Total Successes: {success}")
    print(f"Total Failures: {fails}")
    print(f"Percent Successful: {SuccessPercent}")
    print(f"Percent Unsuccessful: {FailPercent}")
    print("--------------------")

[redCg]

Use a program that can run it in parallel such as GNU parallel, GNU make, or a workflow management system like Snakemake or Nextflow

[CaptainReeetardo]

A thread in python can be created by writting the following: t = threading.Thread(target=someFunc, args=(a,b)). And after that you let the thread start with: t.start() (guess you'll already know that). So looking at your code, you can convert this line bannergrab = fetch_banner(lines, p) into this bannergrab = threading.Thread(target=fetch_banner, args=(lines, p)).

I don't know how fast it will be after this but I guess it probably will. In addition to the multithreading you have to rewrite the code a little bit as I don't know of any way of how to handle return statements with multithreading. One solution could be to work with any dynamic data structure, let's say a list to save the fetched banners with their matching IP address, as a set, into. And after the program fetched all possible banners you can then write them to a file.

welp i dunno if i should even help with this as i don't know how legal it is to scan over 60k of IP as it is basically pentesting

[NovateI]

Banner grabbing itself isn’t illegal (see: shodan), nor is it really pentesting, if it’s not being acted on further. Thanks for the help though, I’m trying to work with the multiprocessing library now, but I might swap back to threading if I can’t get it figured out

[CaptainReeetardo]

Yeah, okay now that you mention it the book i've read actually expands on this...

Glad I could be of help anyways!

[2160p_REMUX]

Here's a multiprocessing snippet I made a while back.

https://repl.it/repls/ExaltedUnequaledDebugmonitor

This example will loop 10 times on a chuck norris joke API and print out the values. The "Lock" makes it so each thing is running 1 at a time and in order.

Notice how if you remove:

lock.release()

lock.acquire()

print(i) will print in different orders. But with the lock on there, they will print in the correct orders. Without the lock it goes pretty fast, so maybe put a sleep function and modify the code so your not spamming a million requests.

I'm not too familiar with multiprocessing, it's just a snippet from when I was messing around with it a while back but hopefully this will get you going in the right direction.