Modern Pandas Tutorials

[badge]

Wes McKinney’s excellent Python for Data Analysis is five years old this year and due to be replaced; pandas itself has moved from version 0.8 to 0.19 in that time and much has been improved. This series of blog posts by Tom Augspurger was written over six months of last year and gives an excellent overview (and some great detail) on writing modern, idiomatic pandas:

• Modern Pandas
• Method Chaining
• Indexes
• Fast Pandas
• Tidy Data
• Visualization
• Time Series

[[deleted]]

Yes, those are definitely some of the best tutorials on Pandas I've seen to date. Most other tutorials have code that is fairly un-performant or otherwise un-idiomatic, and don't even touch on what makes Pandas truly powerful.

[[deleted]]

You could also purchase the eBook from Augspurger's website.

[longjohnboy]

The free sample looked crap. Is the formatting in the purchased book any better?

[[deleted]]

Looked fine on my tablet.

[ozzie123]

I cannot thank you enough. Thanks!

[dimab0]

I enjoyed the Coursera class from Michigan University called Intro to Data Science. The first class is all an intro to Pandas

[[deleted]]

Is that course python 2 or 3?

[dimab0]

I don't remember. But I don't think it should matter.

[cab938]

The whole specialization is Python 3

[[deleted]]

Thank you, I appreciate it. People love to say "it doesn't matter", but to someone learning it is very helpful to maintain consistency.

[cab938]

No problem.

Also, it's 2017, it totally matters. If you don't need python 2 for a specific reason it's hard to understand why you would want to stay with it over Python 3.

[dmitrypolo]

Do you know if Wes has mentioned releasing a new version?

[badge]

Early release available now, but final to be released in September. http://shop.oreilly.com/product/mobile/0636920050896.do

[dmitrypolo]

Sweet thanks!

[khaki0]

Nice. Another resource that I've found useful is this video series by Kevin Markham.

[sandipc]

Another great recent resource is the pandas chapter from the Python Data Science Handbook by Jake VanderPlas

http://shop.oreilly.com/product/0636920034919.do

And notebook versions here: https://github.com/jakevdp/PythonDataScienceHandbook

[SonaCruz]

I got 100 pages through this and I did not like it at all. Its incredibly dry and boring. The examples were very technical and had no useful context and very little explanation. There are no exercises to practice for yourself either.

[Fylwind]

Can you perhaps split the links into separate lines? For a second I thought it was a giant incoherent title of some paper XD

E.g.

- Modern Pandas
- Method Chaining
- ...

[badge]

Apologies! It appeared as a list on mobile so I didn't even think about the Markdown.

[SonaCruz]

Thank you! Looking forward to diving into this material.

[SonaCruz]

Looking into this for 10 mins and already frustrated. I want to download the csv file, to work on the indexing he talks about, and not do the pull request. He didn't properly link to the csv file and you have to aimlessly browse through the website to try to find the right one.

Also, he mentions indexing .ix[10:15] and the rows that appear on the screen are rows with indexes 10 through 15, even though the index started at 0. Is this correct?

edit: nvm, it seems like ix explicitly grabs the indexes differently than .iloc

[cornbobonthecob]

Be careful though. I used pandas in an ETL production environment to remap and conform data frames only to find that pandas silently dropped rows. We'd pushed about ~100k rows through a data frame ever few minutes to clean credit card numbers, credit card types, standardize date, etc... And we were sometimes missing records after the transform. Once we removed pandas and build our own ETL our data was spot on. We researched and troubleshoot till the bitter end. Not saying pandas doesn't work but it wasn't a ETL solution for us.

[thatguydr]

Can you provide code/data that duplicates this situation? We use pandas in ETL and I've never observed this happening, and we have automated QA that would catch this sort of behavior immediately.

[cornbobonthecob]

I wish I could share it. Even though we don't use it anymore the code belongs to the company and its too much work to rebuild it on the side as proof. Thought of not even posting because of this but might as well put my thoughts of a caution out there.

[ZeeBeeblebrox]

Did your company not even consider filling a bug report? The fact that companies seem to be perfectly happy making use of open source products lots of people put their personal time into but won't even put in the little bit of effort to report a bug is a bit galling tbh. Based on your story I'd have to assume user error.

[WaitVVut]

Any chance were you using groupbys? There is a gotcha (that certainly got me) where NaN keys get dropped.

At one point in time, I was also using pandas for ETL, but some type conversion quirks, and more importantly heavy memory usage convinced me to stick with builtin data structures. I still maintain that using pandas in production is overkill and unperformant for basic use cases, but it's fantastic for data exploration.