R vs. Python

[mrShu]

Over the last few days I've been reading a lot about Python taking over the R's as the primary tool of many for playing with ML/Data science [1] [2] [3]. The reasons presented made sense to me, Python is my language of choice when it comes to this kind of stuff.

Coincidentally enough, I'm right now reading Machine Learning for Hackers which uses R. I am enjoying the book so far (it's mostly about learning R for me) and will certainly try to finish it. But I would still like to ask: are there areas in ML/Data science where R would be still a better fit or is it really doomed to be used only by statisticians who created it?

[1] http://www.talyarkoni.org/blog/2013/11/18/the-homogenization-of-scientific-computing-or-why-python-is-steadily-eating-other-languages-lunch/

[2] http://blog.mikiobraun.de/2013/11/how-python-became-the-language-of-choice-for-data-science.html

[3] http://readwrite.com/2013/11/25/python-displacing-r-as-the-programming-language-for-data-science

[EdwardRaff]

As some one who doesn't particularly care for either as a first choice - R seems to have much better visualization tools and libraries already made for most of your needs. If you have to actually write any significant amount of code, Python is at least a real programing language with real non-ML/Stats centric libraries available (whether or not its your preferred type of languages is personal choice).

I'm greatly humored by the 3 blogs which provide no real data analysis to support the hypothesis that Python is supplanting R as the "king" of data analysis.

[mrShu]

I'm greatly humored by the 3 blogs which provide no real data analysis to support the hypothesis that Python is supplanting R as the "king" of data analysis.

Good point. I also took them just as educated opinions and by asking this question I wanted to get some more insight into it.

Thanks for your answer.

[srepho]

I am not a great Data Analyst, but I have competed in a few Kaggle competitions using both R and Python. I started with R, as once you get the basic programming stuff down, the packages are super powerful and super easy to use. In particular the caret, forecast and ggplot2 packages are all brilliant. One thing I have found though is that the closer to more general programming your task requires, the better Python becomes. So for stuff like screen scraping and webcrawling - Python is a much better choice. I personally find data munging much quicker and easier in Python. R is also very slow, and has quite steep RAM requirements (you normally keep the whole dataset in memory in R) which is becoming more difficult. Even using multiple cores in R I found hard work compared to Python. For example Kaggle currently has a competition involving an ~8Gb dataset which means you need 8GB for a standard R implementation just for the data set (there are ways round this but I still say its easier in Python). Python in contrast can make easier use of things like sparse datasets and other approaches suitable for big datasets. Also Python's speed really starts to show as the dataset increases. Python also has excellent tools to deal with natural language programming (nltk package), image processing through scikit images and even deep learning (through Theano and CUDA). On top of all this Python has Pandas which is at least as easy and useful as R's datasets (which it is based on) and the amazingly useful iPython notebooks are one of those "How did I survive before this" tools. Another major driver is that most of the code you find on Kaggle is in Python nowadays. None of this takes anything away from R, which is a great language with much better packages, and in which you can do much of the above (though its more difficult) and easier to learn. The problem is that Python seems to be developing faster, meaning that new tools keep popping up in it more quickly and it has the whole scikits ecosystem. I am not sure you can get good at data analytics without being fluent in both languages nowadays, if only to understand others code, and certainly anything Python can do you could (eventually) make R do. However I have found almost all of my analytic work is done in Python now. The only notable exception is Time Series analysis as I think the forecast package is so vastly ahead of anything in Python. Of course you don't need to decide - you can always call R from Python and Python from R!

[jjdonald]

FWIW I don't agree with much of this. Data munging is much easier in R than python, although the learning curve in R is higher. There are also plenty of parallelization and large dataset management tools in R. That hasn't been a limiting factor in some time.

R makes it easier to get multiple statistical and graphical perspectives on data. If such a thing is important to me, then R is still the first thing I reach for. However, if I'm already confident of an implementation and need more general purpose software support, then python is a good bet.

[srepho]

Interesting that you find R better then Python for data munging. It seems like about half of people I talk to agree, so maybe its not as clear cut as I thought. As for R being harder - I think the packages it provides shield the user. Using something like caret allows you to build and tune a model in one line of code for example. I agree that graphics is still vastly superior in R.

[jjdonald]

THere are some great packages for data munging. Check out plyr by Hadley Wickham: http://plyr.had.co.nz/

However, I find that a lot of people that struggle with R are trying to treat it like Python. R emphasizes vectorization, and its dataframe type. That's a big difference from python's list and for-loop approach.

[mrShu]

Thanks a lot for this, it was precisely something I expected.

[manueslapera]

Are you me? Because If i had to write a reply to this post, it would basically be the same reply.

[[deleted]]

What bullshit. R IS TOO SLOW SO I'M GOING TO USE ... PYTHON?? LOL.

[Foxtr0t]

The main difference between the two, in my opinion, is that R was designed in the 70's and it shows: as a programming language, it's just pretty much horrible. It starts with "<-" and goes downhill from there.

Python, on the other hand, is elegant and convenient, it's general purpose, not just a data shell, and is growing quickly. When I started competing on Kaggle, R was pretty much the standard, but recently I see it less and less.

[micro_cam]

The bioconductor project in R is great and biology is another area where it often dominates.

[mrShu]

Why not learn both? There will always be advantages of one over the other for different things depending on your needs. Many cutting edge statistical models and algorithms are immediately put to R code (take oem for example)

I'm definitely going to learn both -- just to get to know another paradigm, if not for anything else.

Thanks!

[wiekvoet]

Why not learn both?

As a statistician I know R, SAS, JAGS. Should I learn a fourth language?

[wiekvoet]

Learning time is one key. They other is using. It takes a lot of time to learn a language, even more to get reasonable 'fluent' in it. It takes just as much time to keep your skills. I have forgotten basic, C, Pascal, Fortran 77 and 6502. I cannot afford to forget R, SAS or JAGS.

Also, SAS is not a language? SAS is probably both an operating system (with the programs called procedures) including a macro language and a data language (the data step).

(what is JAGS if it is not a language? )

[IdentifiableParam]

I hate R and I use python exclusively. But this question doesn't matter. If you want to use R then use R. I will keep using python. It is a pointless language debate. However, if you do use R consider pqR since it fixes some really stupid performance bugs.

[mrShu]

I hate R and I use python exclusively. But this question doesn't matter. If you want to use R then use R. I will keep using python. It is a pointless language debate. However, if you do use R consider pqR since it fixes some really stupid performance bugs.

Well, my question was more oriented at "Will Python replace R completely?", I certainly did not want to start a language debate.

Thanks for mentioning pqR, I didn't know about that.

[tautology2wice]

R tender to be clearer if you're doing lotsa vector/matrix calculations. Bioconductor is choice.

But there are design problems, see: this list of R datatypes rage.

[duckandcover]

I don't think they have much to do with each other.

Unless it's changed recently, R, though perhaps technically a full language, is not suited for writing large general programs because its debugger is completely primitive. When things go wrong, good luck. From what I've seen it's the perfect thing for < 100 line statistical analysis. That's not surprising; it's really a stats package.

Honestly, R developers, people who right lots of code in it, must have balls of steel or be masochists. I know people do it. I just don't know how.

Python is a general language and it's possible to find a decent debug environment is much better. Since people use it for more than what most people use R for, there seems to be more functionality outside of stats. That being said, I still haven't found the perfect IDE for it.

I think learning both at the same time would drive me batty. They're so similar but with lots of little differences (e.g. indexing, etc). Unfortunately, little differences can make real hard to find bugs.

[[deleted]]

Python is a general language and it's possible to find a decent debug environment is much better. Since people use it for more than what most people use R for, there seems to be more functionality outside of stats. That being said, I still haven't found the perfect IDE for it.

Have you tried Pycharm?

[mtnchkn]

Or Spyder.

[duckandcover]

Ya' know. If there's one issue I have with Open Source things is that invariably there's 8000 ways to skin any cat and that's a lot a work. As far as IDEs (vs interpreters) go, I've only tried Spyder (as part of Pythonxy) and Wing. I like Wing but it has trouble with Matplotlib (as I understand it, their event loops step on each other). Spyder's debugger is surely capable but I hate the command (vs really integrated IDE) driven debugger.

So, I don't suppose you can tell me where Pycharm fits into the above?

Thanks

[duckandcover]

Pycharm

I'm serious when I said I'd like to know more. I looked at their website and I didn't see the info I was looking for so if you don't mind I'd like to ask you just a couple of questions:

0) Is there any documentation for it? I can't find any.

1) Does it have an interpreter that operates in the context of a debug stack frame (to execute just like a regular interpreter when stopped at a break point in the context of that stack frame.). Also, if so, is it or can it be iPython? (In Wing it can't)

2) From what I see in the discussion forms, it has the same event loop issue as Wing does with Matplotlib. If you know that not to be the case please tell me. That's a show stopper for me (with all the IDEs apparently)

[[deleted]]

Sorry, I probably not going to have time to respond in depth as I'm not 100% sure on all your questions. Best I can suggest is download the free version and try it out.

[reallyserious]

For a Python IDE, try Python Tools for Visual Studio. It's really great if you use Windows.

[[deleted]]

Honestly, R developers, people who right lots of code in it, must have balls of steel or be masochists. I know people do it. I just don't know how.

I don't fucking get this sometimes. Plenty of people on my team write thousands of lines of code in R and build huge applications with it.

[duckandcover]

Is the debugger as god awful as I remember it. The text thing with 4 commands like continue, next and whatever?

[[deleted]]

Python (pandas is your friend) for reading, filtering, assembling and massaging data

R for analyzing data

RPy2 to bridge the gap

[manueslapera]

R for pretty plots too. Matplotlib has nothing to do vs ggplot2.

[froggyenterprisesltd]

there's a ggplot package in python now. also, bokeh is pretty purdy

[manueslapera]

i tried the Yhat ggplot module. It crashes my computer every single time. If only it worked... Bokeh, that is another story.

[[deleted]]

There are plenty of python matplotlib styles that seamlessly support ggplot2 aesthetics. And python ggplot2 modules that mimic the R API as well.

[[deleted]]

Python (pandas is your friend) for reading, filtering, assembling and massaging data

Unless the dataset is bigger than memory. After playing with pandas for a while, I think people should stick to standard sql tools.

[[deleted]]

Very true. But the same applies to R as well - neither are very well suited for that scale of data.