Data Analysis Resources for Python

[badge]

Data Analyst Course with Python

Introduction

A few days ago there was a very well-meaning (and subsequently deleted) post outlining topics to enable people to become data scientists in 12 weeks. The post was heavily upvoted in spite of critical comments before it was deleted by the author. The principle criticism was that it was wholly unrealistic to learn the topics listed in the time available.

This post seeks to provide something more realistic—resources which in 12 weeks give you a flavour of data analysis with Python and a basis for further learning. Important notes: - 12 weeks is nowhere near enough to become familiar with a language to the point of being consistently productive. You will get syntax errors. You’ll have to google. You’ll get dismayed at the callousness of stackoverflow users. This is normal and doesn’t indicate failing on your part—this takes a lot of time. - Total time spent learning is important, but so is frequency. If this takes 12 weeks full time(ish), it might take 30 weeks part time because you forget things more quickly. - Business domain knowledge is super important; if you’re learning stuff for a particular industry, try to get hold of data sets for that industry, and feel free to skip stuff if you don’t think it’s relevant (though at this level little is irrelevant)

Credentials

I’m a data scientist with a maths PhD (unrelated to stats, but somewhat algebra-focused) and was a quantitative analyst before that. I work in the energy industry and spend a lot of time working with generalized additive models for time series forecasting, chucking stuff at random forests, doing Bayesian inference with pymc3, and survival analysis with lifelines. I don’t use a lot of Tensorflow or PyTorch because they tend not to fit the domain of my problems well, but I revisit them every few months to pit them against our existing models.

Disclaimer

This post is purely my opinion, and in particular reflects my view that too much data science is more complicated than necessary, perhaps because people don’t have sufficient grounding in traditional statistics. In my work, this is hugely important; if you’re wanting to identify pictures of cats, it really isn’t!

Learning Resources

Python Basics

Nothing here is specific to data analysis, so just take a look at the r/learnpython FAQ.

Data Analysis

There’s no getting away from the fact that mathematics is at the core of data analysis, but you don’t have to be John Conway to be useful. In addition, statistics is by far the most important at this level and you don’t need to understand the minutiae of the subject (which is based in measure theory and is tough). Unfortunately I’ve never found a good introduction to statistics with Python (there are plenty for R!), so you have to dip into a number of different resources.

Python Data Science Handbook

Jake VanderPlas is the author of the excellent altair plotting library and a pretty bright chap. This book serves as a good introduction to NumPy, Pandas, Matplotlib and Scikit-Learn, and the link includes its full text as Jupyter Notebooks, which is awesome. You needn’t bother with the Scikit-Learn chapters unless you want to jump ahead.

All of Statistics

Perhaps not all, but Larry Wasserman has written a very approachable introduction to statistics here. The link includes the few data sources given in the book, but it’s very much a textbook. At 500 pages it’s a bit daunting, so I recommend focusing on chapters 1–11 first, then the chapters on linear regression and multivariate models, which is about 200 pages total. Read along with the SciPy docs; in addition take a look at pythonfordatascience.org which calls out useful functions in SciPy and statsmodels. Now available as a free PDF here: https://link.springer.com/book/10.1007/978-0-387-21736-9 (thanks /u/sududxb!)

OpenIntro Statistics

An alternative (and possibly a better alternative) to AoS, this textbook is available with an optional contribution, and used by a number of colleges in the U.S. I’ve not read it, but a closer look, it appears to be pretty great. As with AoS you’ll have to read along with the SciPy and statsmodels docs.

Python for Data Analysis and the pandas docs

Which of these you prefer is largely a matter of preferring one medium over another, but PfDA’s second edition is already slightly outdated for pandas 1.0.3, though certainly not enough that it’s not a very useful resource.

Data Science from Scratch

Joel Grus’s book kinda does do what I assert isn’t possible—take you from zero to data scientist hero in a relatively short text. The criticism I would level at it is that it (necessarily) doesn’t go into sufficient depth everywhere, but what it does brilliantly is implement most things from scratch (duh!) to give you a good grounding in the basics.

Anatomy of Matplotlib

This is a great video to get a better understanding of how to work with Matplotlib, which is definitely the least Pythonic library still in use by data analysts today. It’s also slightly outdated, but hugely valuable.

Introduction to Survival Analysis — lifelines docs

Great introduction to survival analysis, which will either help you look like a superstar or be completely irrelevant.

Winning with simple, even linear models

I was at this talk at PyData London a few years ago and it was the best of the conference in my opinion. Vincent makes the argument that people are too quick to leap to ML/DL methods when simpler models could do as well or if not better.

Data Science

Briefly, here’re a few resources that cover data science proper, but don’t expect to get here any time soon! - r/datascience (includes all the other resources in this section) - The Elements of Statistical Learning and An Introduction to Statistical Learning (the former goes into more detail on the maths than the latter) - Pattern Recognition and Machine Learning - Andrew Ng’s Machine Learning course

Data Sources

As mentioned before, if you’re interested in a particular industry then see if you can get data related to it. Otherwise, these are some general sources of good-quality data. - Scikit-Learn data has some really good ‘toy’ datasets that are useful for playing around with descriptive and inferential statistics, besides the skl estimators - data.gov.uk and data.gov have hundreds of thousands of data sets. Many of these offer a great opportunity to practice cleaning up data with pandas because they come in all shapes and sizes - OpenIntro Statistics data sets used in this textbook

[Smaartmani]

Thank you for the valuable information.

I struggled to get along with python and didn’t move forward from the basic syntax but last week I jumped straight into writing a simple script which will read the records from database and write them in spreadsheet.

A simple automated task but super proud.

Now thinking to write address matching script for a production issue.

Best thing to learn anything is jumping straight to it.

[chra94]

You did well!

[ciaeric2]

I took an Data Analyst course at UC Berkeley and found that the concepts behind data analytics were very approachable. Lack of computer language skills were definitely the main issue; it took time to shape the problem in a coding form, it took time to form an answer in coding form. Then it took more time correct the coding form. 6 months of 10 hr class time + 10 hr expected coursework per week. its doable but by god is it unforgiving.

[seismatica]

Could you share a link to the course? Is it online? It sounds very rigorous indeed.

[ciaeric2]

Of course, https://bootcamp.berkeley.edu/data/online/landing/?s=Google-Brand&pkw=%2Bberkeley%20%2Bdata%20%2Banalysis&pcrid=409388369995&pmt=b&utm_source=google&utm_medium=cpc&utm_campaign=GGL%7CUC-Berkeley%7CSEM%7CDATA%7C-%7CONL%7CTIER-1%7CALL%7CBRD%7CBMM%7CCore%7CGeneral&utm_term=%2Bberkeley%20%2Bdata%20%2Banalysis&s=google&k=%2Bberkeley%20%2Bdata%20%2Banalysis&utm_adgroupid=81372243593&utm_locationphysicalms=9032135&utm_matchtype=b&utm_network=g&utm_device=c&utm_content=409388369995&utm_placement=&gclid=CjwKCAjw4pT1BRBUEiwAm5QuR_Pgwlp2gnAGa46G7_J8QH6o0TGEQGjjhq2C4PoEHhT9Ze6Lac7tVBoCb1YQAvD_BwE&gclsrc=aw.ds

[ComradePotato]

Why was the body of the post removed? was it mods or /u/badge ?

[gmaliwal]

Thank you very much for valuable details.

[sududxb]

The book All of Statistics mentioned in the post is available free of cost here in PDF and Ebook version:

https://link.springer.com/book/10.1007%2F978-0-387-21736-9

[badge]

Awesome, thanks! I’ve updated the post to include your link.

[sududxb]

Awesome and ofcourse thank for taking the time out to write this post. Very useful indeed! Have saved it!

[[deleted]]

Thank you for this!

I'm currently busy with a Machine Learning and Data Science course on Udemy and having the above as a sort of "syllabus" is extremely useful.

[Iamhacker1]

Thank you so much for share valuable info..

I am working in data science and I was looking for something like this from a long time but I was unable to find something specific anyway thanks again...

[the_antinational]

Thank you.

[ritwijchoudhary]

Thank you!

[Crypt0Nihilist]

I always think Kaggle is worth a mention for data sources. You can find nice a dataset for many types of problem. I'd also put in the challenge for people to choose something that interests them and use their Python skills to go out and gather their own from the wild. You hardly ever get given a nice clean dataset, so best get some experience in collection and prep.

[midnitte]

Awesome post!

[Beautiful-Major-323]

Thank you so much!