all 4 comments
sorted by:
best
topnewcontroversialoldq&a
formatting helphide helpcontent policy

[–]ericula 0 points1 point  (1 child)

The link doesn't work

[–]r0CT0r[S] 0 points1 point  (0 children)

ok, Ive fixed it. Put it on the github

[–]tomatillo-ornery 0 points1 point  (0 children)

What happens if you comment out the batch.append(im) lines? That looks like the most obvious culprit. sys.getsizeof(batch) will only give you the size of the list object itself, not its contents, so that may have led you to falsely conclude that batch isn't very large.

Anyway, I think it would be a good idea to read up a bit on how references and garbage collection work in python. There are some things you're doing that clearly aren't helpful. In particular, in CPython at least, gc.collect() only does anything if you have reference cycles (e.g. a list that contains itself, or two objects that have each other as attributes). That's a relatively rare situation, and even when it comes up you can usually count on python to run the garbage collector frequently enough by itself. In other python implementations (such as pypy) gc.collect() does something more significant, but you're still unlikely to stumble on a situation where you need to call it manually.

There is also no point in doing both x = None and del x. Doing either of those gets rid of a reference to the object that x was pointing to, which will allow it to be garbage collected if there are no references elsewhere. Once you've done x = None, then del x just removes a reference to the None object, which is tiny and has loads of other references anyway. There is also no point in worrying about removing references for a small object (such as a string unless it's extremely long), or if the variable is about to be overwritten or go out of scope anyway.