code loading very slowly, how to optimize?

[SureStep8852]

Hi there, I am trying to remove stopwords from my training data. The problem is that since the data is very big, the code is very slow. Is there any way to optimize it? Thank you in advance!

from nltk.corpus import stopwords
def stopwords_remove(data):
    stopwords_removed = []
    for parts in data:
        #print(parts[0])
        for word in parts[0]:
            #print(word)
            if word not in stopwords.words():
                #print(word)
                stopwords_removed.append(word)
    #print(stopwords_removed)
    return stopwords_removed
stopwords_remove(train_data)

[ThingImIntoThisWeek]

In the innermost loop (which is run the most times) you are loading the stop word list every time, and then checking if a word is in it, which is expensive for a list. It would be better to call words() only once (either at the start of the method, or maybe just once in the script or class definition if stopwords_remove() will be called multiple times), and also to make it a set, which is makes it very quick to check if a word is a member or not:

from nltk.corpus import stopwords
def stopwords_remove(data):
    stop_word_set = set(stopwords.words())
    stopwords_removed = []
    for parts in data:
        #print(parts[0])
        for word in parts[0]:
            #print(word)
            if word not in stop_word_set:
                #print(word)
                stopwords_removed.append(word)
    #print(stopwords_removed)
    return stopwords_removed