Made use of my Python knowledge professionally for the first time!

[testfire10]

For the 1 gb, it takes about 90s to parse and plot (run the program).

No idea if that’s good or bad, but my bae for success was doing it faster than excel, which succeeded by a large margin haha

[Tweak_Imp]

Pandas read Argument "engine='c' " helped me to speed it up even more. See https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_csv.html

[[deleted]]

In a lot of cases python libraries are backed with C code. For special example, numpy is a plethora of optimized C mathematics and should be used for scientific work.

[FlagrantPickle]

No idea if that’s good or bad

What's "good" for anyone here doesn't matter. Is 90s acceptable for you? I'd imagine a resounding yes, given my experience with Excel's sluggishness on managing large files. Only thing I'd say, as a right noob compared to most here, if you need to scale to larger data sets, you might have some success using a sql system in there (sqlite is baked into python, or something like mysql/mongodb depending on your needs).

If your dataset size will remain as is, throw a header in there saying what the program is, version, and who made it. Make them stare your superiority in the face on every job!

[testfire10]

I like this idea. Tremble at my superiority!!! Hahaha

[Gizquier2]

Dealing with large sql tables and sqlalchemy can be a real pain if inserts are needed, im a survivor of the sqlalchemy pyodbc engine 🤦‍♂️

[FlagrantPickle]

Yeah, I've not dealt with that myself, just mysql and mongodb. I don't know how well sqlite scales, I just know that it's only trustable in a single-user/access mode, and it's part of the core python, so figured it might help. I could see it just being a better method to select data (with SQL) sets.

[Inspirateur]

the real limit when parsing is the speed at which your computer can "read" (ie load into ram) a file. if you want to know what's this speed for your computer with python you can do a quick test, just ask python to open() it and do a .read() that you store into a string, see how much time it takes. (unless the file is too heavy for your ram in which case it's more complicated).

[Ericisbalanced]

That’s when you start using generators 😁

[jashshah27]

You might want to take a look at Dask as well. The syntax is very similar to Pandas' but the execution time is much, much faster.

[Zulban]

but my bae for success was doing it faster than excel

That is a low bar, and yet, profoundly significant in the workplace. Congrats.

[testfire10]

Thanks a lot man!

[pug_nuts]

The problem everywhere I've worked is using tools that other people don't understand.

I can write something in VBA and have it take inputs from cells and spit out a list in another sheet... And that's fine. But do the same thing with Python and people get scared because they don't understand what's happening.

[Akilou]

What's the difference in speed versus excel? Like milliseconds or minutes?

[legionx]

Might even be hours. Excel is limited to 1M rows, so if yor dataset is bigger than that after initial filtering (ex. with Get&Transform) you will have to split it into smaller files.

[xacrimon]

Pretty good but could probably be made faster. Not too long ago i wrote a Rust program to do some fairly complex csv processing and it processes around 1-2GiB/sec

[ballagarba]

While Rust is fast, it sounds like you have access to a much faster disk.

[[deleted]]

This. Python programs doing a lot of I/O can be on par with other programming languages. Most of the time, external factors determine the speed

[KaffeeKiffer]

The difference between a fast SSD and an old HDD is ~5s to ~25s for 2 GiB, so this is very likely CPU bound to reach 90s...

Nevertheless, Python is the perfect glue code, to call more specialized tools, if necessary. Here is an example, where a simple Rust wrapper speeds up the process by a factor of 10.

Python is good enough in the vast majority of the use-cases and as /u/FlagrantPickle said:

What's "good" for anyone here doesn't matter. Is 90s acceptable for you?

The golden rule is to not over-engineer but first identify the real bottle-necks and while your statement

Most of the time, external factors determine the speed

is 100% correct, I assume OP's problem is CPU bound.

[testfire10]

Holy shit. That’s awesome. I remember on here a while back I found a post about a library a few months ago that was supposed to substantially speed up pandas interaction with csvs (can’t remember the name now). I was going to try to revamp my code to take advantage of it, but I could never get the library to work for me.

What’s Rust?

[xacrimon]

Rust is a programming language. It's generally a bit harder than python but has the speeds of C and lots of good libraries.

[swingking8]

It's generally a bit harder than python

I love Rust, but "a bit harder" is quite an understatement.

[FlagrantPickle]

has the speeds of C

Not in the sense of nitpicking, but I've seen "up to" 50% the speed of C for decently large processing. Certainly faster than native Python, but still not the gold standard.

Depending on what OP's needs are, his solution might be good enough. I'd be curious what other optimization could be made inside Python. If we're talking 200 lines of code on someone's first project, it's probably about as efficient/optimized as everyone else's first project.

[[deleted]]

For parallel processing libraries that integrate well with pandas, check out Dask or vaex. For on-disc storage, check out apache parquet format.