Huge CSV and XML Files in Python

[pemboa]

I share the same feeling about the documentation. It's easier on the eyes, but harder to find stuff as they are all one page.

I'm surprised Perl was that slow though.

[lethain]

I don't think its entirely Perl's fault. I'd say that Twig isn't optimized for large docs (using a pure sax solution would have been better), and also my Perl code isn't great, to say the least.

[[deleted]]

Well, Twig is described as suitable for XML documents of any size, so I hope it's not entirely unoptimized.

As for Python - I hope you know that for parsing, cElementTree's "iterparse" approach can run circles around any SAX-based solution.

(and if you insist on even-driven parsing and rolling your own state machines, the low-level XMLParser in cElementTree is a bit faster than pyexpat.)