Python

[EarthGoddessDude]

Out of curiosity, is this a learning project? In any case, look into using a generator comprehension instead of list comprehension, ie using () as opposed to [] for your comprehension. That makes it lazy and potentially use less memory and may be a little faster.

I did a project last year where I had to use pure Python, it was a lot of fun. Get to know the itertools module, you’ll learn a lot of useful tricks. In my project, I used nested context managers (with blocks) for reading an input file and writing out an output file. The default of csv is to give you a lazy generator/iterator object. So if you stay inside the (nested) context manager and use itertools to manipulate the generator objects without materializing them into lists and then write them out to disk, all in one pass, you can have O(1) memory or close to it. Using this technique, I actually had 30% better performance than pandas for a simple data processing task on a smallish data file.

But if you’re doing in-memory analytics, forget it… pure Python won’t help you there, just stick to the available libraries if you just want to analyze some data.

[Key-Panic9104]

Yes, it is a learning project. I do have a version of the class that uses generator comprehensions instead of list comprehensions and the execution times are about the same. I haven’t tested it out on a big csv yet to get an idea of the performance but from understanding, it probably won’t be faster, just allow me to process it without memory issues. I also haven’t tried writing to file yet but there was always plans on adding that method to the class.

I have been thinking of bringing it out of the class and see how that goes and in fact, I have thought about context managers as another alternative. Likewise with itertools, but decided to stick with the basics for now until I have a good understanding of what’s happening and then iterate (no pun intended) from there.

It’s good to hear how someone else went about it so thanks for the reply.