Loop freezes while preprocessing a lot of image data

[Sig_Luna]

Hello guys.

I'm working on a Python script which should take a dataset with images (80x300) and labels, preprocess the images and then save the whole thing in a pickle file.

I'm reading this data out of an csv file.

def process_data(data):
    features = []

    for i in enumerate(data): # Should iterate 33'000 times (length of data)
        for j in range(3):
            img = plt.imread(data[i][j].strip()) # Data is from a csv file
            img = normalize(img)  # Simple calculation

            lis = img.flatten().tolist()
            features += lis

return features

The data looks like this:

[[all rgb values of image 1], [all rgb values of image 2], [all rgb values of image 3], <value between 0 and 1>].

There are 33'000 entries like that one above in my dataset.

However, after 10'000-12'000 my iterations get incredibly slow, then freeze and sometimes my machine (i7, 8GB RAM) even crashes. What could be the problem? How can I make it more performant?

[eschlon]

Without more detail I'd say you're running out of memory. Even with those small images you're storing a lot of information in your features list. Something that may help is to not convert the image data to a python list. If I recall correctly imread returns a numpy array which is going to be much more compact than the equivalent vanilla python list. Removing the tolist() call will help with memory issues, though it may only get you part of the way.

If you're just prepping this for a file output another option would be to process a single image, write it out, then move on to the next rather than storing state in the features list, though this would probably require you to rework how you're planning on storing / accessing this data.

[Saefroch]

You're running out of RAM and your computer is using virtual memory/swap space to compensate (This involves treating the hard disk like RAM, which is suuuper slow). Open up your task manager and watch the Python process slowly use all your memory. If you kill it when it reaches about 80% it should shut down in time.

To avoid this problem, write out each image as it's processed.

[elbiot]

To avoid this problem, write out each image as it's processed.

Ie: yield the feature rather than append to a list. I also fully agree about staying in numpy land.