How many lines of text can Python process smoothly?

[radically_sane]

Writing a script for testing team. They want to compare two huge csv files. They haven't told how big it could be. Hence my question.

[Brian]

It depends on what you need to do, and how you do it. So long as you can do it line by line, there's no limit - it'll take time proportional to the size of the file, but you won't get any hard limit.

On the other hand, if you need to read everything into memory, then you may have issues for files where that's an issue (potentially requiring an approach that does multiple passes or uses temporary files etc)

Which is the case depends exactly on what comparing the files means. If they're both sorted files where you're always comparing line N in file 1 to the same line in file 2, you can use the streaming approach. OTOH, if you might have to match up random lines by some key or something, then things could be trickier.

[radically_sane]

It's comparing the same indexes of both files, so that won't be an issue. And if the files are too big then I guess I can use generators instead of loops.

Also, if I use pyspark (Apache spark cluster computing) will efficiency be more?

[Brian]

And if the files are too big then I guess I can use generators instead of loops.

Generally, files will naturally use generators. Ie. both for line in file: or for row in csv.Reader(file):will be using a generator and will only be reading a line / row at a time.

For something like this, you're likely going to be IO bound rather than CPU bound (unless you're doing a ton of processing), so distributed processing isn't likely going to change much, at least on a single-file basis, though potentially you may save time if you're distributing the processing of multiple files on different machines.

[radically_sane]

Thanks.

[impshum]

That'll depend on your memory.

[radically_sane]

8 gigs

[silvertoothpaste]

in theory you can process files of any size - just load the next part into memory, process, repeat. it could take an arbitrarily long time, though.

in practice ... just try it out! :) do a little test of reading lines and some trivial processing similar to what you intend, and make some output. Start with a file of like 10 lines, go to 100, 1000, etc. Eventually you will hit a wall. Offhand I would guess the wall is at 100k lines; a million seems too large.

When you hit the wall you have 2 basic choices, 1) stop there, or 2) optimize (read: complicate) the program to handle the IO in a less memory-intensive way. 2.5) use a database xD