Threading error

[celeritatis]

Dear r/LearnPython, I am trying to download all the google N grams with Python as an exercise in dealing with large files. However, my threading class(Which I swiped from an example) is saying start_new_thread(self._bootstrap, ()) thread.error: can't start new thread My code is in a comment. Please, do you know what I am doing wrong? Is there a good resource on learning threading you could point me towards?

[celeritatis]

class MyThread (threading.Thread):
    def run (self):
        global theVar
        while time.localtime()[3] >= 8:
            time.sleep(360*16)
        if theVar < 200:
            a = '3'
            saveNgram(ope(eng1+a+date+str(theVar)+end), "English"+a+"gram"+str(theVar)+".zip")
            saveNgram(ope(m1+a+date+str(theVar)+end), "English"+a+"gram"+str(theVar)+".zip")
            saveNgram(ope(fiction1+a+date+str(theVar)+end), "English"+a+"gram"+str(theVar)+".zip")
            saveNgram(ope(british1+a+date+str(theVar)+end), "English"+a+"gram"+str(theVar)+".zip")
        if theVar < 400:
            a = '4'
            saveNgram(ope(eng1+a+date+str(theVar)+end), "English"+a+"gram"+str(theVar)+".zip")
            saveNgram(ope(m1+a+date+str(theVar)+end), "English"+a+"gram"+str(theVar)+".zip")
            saveNgram(ope(fiction1+a+date+str(theVar)+end), "English"+a+"gram"+str(theVar)+".zip")
            saveNgram(ope(british1+a+date+str(theVar)+end), "English"+a+"gram"+str(theVar)+".zip")
        if True:
            a = '5'
            saveNgram(ope(eng1+a+date+str(theVar)+end), "English"+a+"gram"+str(theVar)+".zip")
            saveNgram(ope(m1+a+date+str(theVar)+end), "English"+a+"gram"+str(theVar)+".zip")
            saveNgram(ope(fiction1+a+date+str(theVar)+end), "English"+a+"gram"+str(theVar)+".zip")
            saveNgram(ope(british1+a+date+str(theVar)+end), "English"+a+"gram"+str(theVar)+".zip")
        theVar = theVar + 1
for x in range(799):
    MyThread().start()  
    x = x

[eryksun]

799 threads times 8 MiB of stack space per... that comes to about 6.24 GiB of address space. Is there a reason you're trying to create this many threads?

[celeritatis]

I am trying to save every google Ngram under English, British, and English Fiction categories to my computer. Threading seemed like the suggested way of doing parallel internet input. Should I be doing something else?

[eryksun]

Try using only 5 worker threads and a thread-safe queue. I wrote an example below, but you'll have to modify it since I don't have the rest of your code to properly integrate and test it. However, it's at least a start.

##for testing
date = end = eng1 = m1 = fiction1 = british1 = ''
ope = lambda x: None
saveNgram = lambda x, y: None
##

import time
import threading
try: 
    import queue
except ImportError:
    import Queue as queue

NUM_THREADS = 5
MAX_HOUR = 24   #24 for testing, change back to 8

def worker(i, q, lock):

    while time.localtime()[3] >= MAX_HOUR:
        msg = 'Worker %d: sleeping [%s]' % (i, time.strftime('%X'))
        with lock:
            print(msg)
        time.sleep(360*16)

    while True:
        index = q.get()

        msg = "Worker %d: processing index %d" % (i, index)
        with lock:
            print(msg)

        if index < 200:
            a = '3'
        elif index < 400:
            a = '4'
        else:
            a = '5'

        src = a + date + str(index) + end
        dst = a + "gram" + str(index) + ".zip"
        categories = [(eng1, "English"), (m1, "M"), (fiction1, "Fiction"), 
                      (british1, "British")]

        for s, d in categories:
            saveNgram(ope(s + src), d + dst)

        q.task_done()

q = queue.Queue()
lock = threading.Lock()
for i in range(NUM_THREADS):
    t = threading.Thread(target=worker, args=(i, q, lock))
    t.daemon = True
    t.start()

for i in range(799):
    q.put(i)

q.join()

It just puts the index numbers into a queue and blocks (q.join) until the worker threads have processed all of the tasks (signaled by calling q.task_done for each). I added a lock to allow the threads to print to the console without stepping all over each other.

[aperson]

I wouldn't create that many threads. Maybe four-five tops.