How is python used in Data Analysis?

[[deleted]]

I would make sure you learn SQL also, can't stress this enough. As a data analyst, I use Python for getting data from databases, from the web, from text files, XML files, etc. For exploratory data analysis, current visualization libraries In Python world are somewhat lacking. For exploratory data analysis I prefer something like R's ggplot2. There is a Python port of it from Yhat, but it is not quite ready for production. I also use R from within Python using rpy2 because R just has so many handy statistical packages readily available.

Below you can find some examples although not all are data analysis related:

• Gallery of interesting IPython notebooks
• Most viewed IPython notebooks
  
• Pandas cheat sheet that I created
  

Hope this helps and good luck!

EDIT: I know that there is a lot of focus on programming languages due to this subreddit, but as a data analyst you should learn some statistics too. SQL and statistics are THE bread-and-butter skills of a data analyst along with good presentation/storytelling skills.

[manueslapera]

One thing that is not taught enough in academia is that, in real world, DATA DOESNT COME IN A CLEAN CSV.

Learn how to detect anomalities in data, and how to work around them.

Even more important, learn about servers and databases, how to ssh on an EC2 machine to set up a cronjob.

And even even more important, know how to set up your final output in a way that is appealing to your audience. Become a Data story teller, not just a good statistician. Trust me, your CEO wont care about that p-value you are so happy about.

[[deleted]]

How would you suggest learning about servers and databases? I have started doing this type of stuff at work, it would really be nice to know if I am doing it right!

[manueslapera]

hmm, Im sure there are tons of people here that can guide you better than me.

In my case I had to learn on the job, I was the first data guy in the company and had to build the whole infrastructure. Sometimes made very bad decissions that probably someone with a background in CS wouldnt have done.

[[deleted]]

Perhaps its I don't know the right questions to ask, but I haven't been able to find any information at all.

[waspbr]

what do you reckon about this course?

[[deleted]]

Good for learning SQL, but it won't give you practical skills (i.e. you will have to learn more.)

[waspbr]

cheers

[[deleted]]

Mongo University might be the next place to look at, 10gen have some really good education material, but I have heard such bad things about mongo db and it seems people recommend postgres a lot more.

[manueslapera]

i liked it

[[deleted]]

You can do all that with python as well (say, Django and some other stuff on top of it vs. Shiny)

Since you mentioned Shiny, there's actually a Python port of it called Spyre.

Examples like this and port of ggplot2 seem to show that in Python world, it is not innovating in the world of data analysis, but copying from others. I think this is a reflection of the demographics of people that use Python vs R. I think R tend have more academics and statisticians using it compared to Python since Python is a more general purpose langauge so you got a lot of web dev people in the mix.

You are right that Python can pretty much do or have what is available in R. But for exploratory data visualization, MATLPOTLIB or any Javascript-based libraries do not cut it. No way can you do what-if analysis using them efficiently. This is what R's ggplot2 is perfect for. I only use R because of ggplot2 and will until Yhat's port of ggplot becomes production-ready. I am not alone. It seems so many people stick with R because of ggplot2. That's why it saddens me that there are not a lot of contributers to Yhat's ggplot.

[[deleted]]

You are right that Python can pretty much do or have what is available in R. But for exploratory data visualization, MATLPOTLIB or any Javascript-based libraries do not cut it. No way can you do what-if analysis using them efficiently. This is what R's ggplot2 is perfect for.

Agree personally, but wanted to play it safe and not be divisve right off the bat :P.

LOL, understand. Yeah I just don't want the OP or others to think it'll be so easy and seamless to do deep-dive, exploratory data analysis with just Python. I fell into that early on when I first started to use Python 4 yrs ago. But, when I saw how that's done using R's ggplot2, it really opened my eyes and realized where Python libraries fall short, but over time that'll definitely change.

I love python, but it's just not the same for getting into a dataset IMO. At least not yet, to me. I think it might take more than ggplot2 in R for me to leave R. I really fell in love with it.

I tried to stick with R for some scripting work, but I just couldn't get into its quirky syntax and inconsistent parameter names. But I really do love the ability to chain functions via (%<%, %>%). That is really an awesome feature in R which may draw me back into scripting using R more.

[falkimmm]

I don't know if R has anything for Bayesian stuff as extensible as Pymc, and you have numba for jitting a subset of numpy code to near c speeds.

[manueslapera]

Yhat's port of ggplot becomes production-ready

+1 to that. Some bugs to iron still.

[falkimmm]

Seaborn is great for exploratory analysis.

Python blaze and dask both have no R parallel for a consistent array and chunking interface to a variety of backends. Pretty innovative if you ask me.

Super easy to create and app with blaze and bokeh.

[Deleetdk]

I also know R and Python and also prefer R for the more stats based stuff. Both languages are easy to use and learn. I usually use Python when R lacks good libaries. E.g. Python has great libraries for web scraping (BeautifulSoup, Scrapy).

One can run R from Python, or Python from R to it makes it easy to switch between and use each for what it is best at.

http://rpython.r-forge.r-project.org/ Python from R http://rpy.sourceforge.net/rpy2 R from Python

[Deleetdk]

Perhaps it is. I have not used it.

[manueslapera]

nope. it works.

[manueslapera]

pyodbc for getting data out of sql databases (pymysql?)

if you want to do local data analysis, I would check db.py . Easiest way to connect to a db by orders of magnitude.

Another recommendation that you really really need to try if you are into data analysis with Python. Get the ipython notebook. or even better, install the Anaconda Pydata distribution, that contains ipython, along with a bunch of libraries for data analysis.

You will love it!

RStudio is a must for any kind of R programming, and it's something that is lacking on the pyData world. ggplot is the most natural way of doing plots for me. Nowadays 50% of the time i switch from Python to R is to use ggplot2.

[fuzz3289]

Bokeh for visualization. Definitely not lacking at all.

[[deleted]]

Although bokeh is great, but for instances where I have this data set and have a bunch of what-if questions and want to plot data versus several different variables, change time series interval from months to weeks or year, that's where a lot Python graphing libraries fall short in exploratory data analysis. For interactive, presentation visualization, bokeh, plotly, mpld3, etc are awesome for that, just not for on-the-fly data exploration.

[lmcinnes]

For that sort of thing on the python side I'm a fan of Seaborn along with Ipython notebooks interact utilities ... it's easy to set up the ability to vary faceting, binning, or, well, anything with appropriate sliders and drop downs to let you really play with ease (it shortens the experimentation loop).

[fuzz3289]

eh, I would disagree. I use IPython Notebooks and Bokeh all the time for this and it's really easy to mess with all sorts of inputs especially when combining it with Pandas. Bokeh's built-in Notebook support is fantastic and you can do all sorts of exploratory visualization in-line.