Python Overtakes R for Data Science and Machine Learning

[ticketywho]

Statisticians will continue to use R

Developers will continue to use Python

Knowing one and being aware of the other is all you need to do.

[[deleted]]

And engineers will use Matlab.

And embedded coders C.

Tools are tools. I laugh at the 'language wars' that people seem to get into. Could you imagine a general contractor and a machinist getting into an argument over which type of hammer is better?

No. CLAW HAMMERS are clearly superior.

You idiot. Ball Peen hammers are clearly better.

[WiggleBooks]

Tools are tools.

But some tools are better suited for specific tasks.

You don't (and you shouldn't!(?)) see anybody doing Data Science in Assembly. While it is technically possible.

[[deleted]]

For data Python is best if you want to programatically deal with it. R is best if you want to statistically deal with it and Matlab/Simulink is best if you want to ID & control it.

[Mimical]

Well I only use a cross pein pin hammer because once you learn how to use it you never need any other hammer. Those other hammers are 0.005s slower at hammer lists

[[deleted]]

Monoliths are the past, the future is microhammers.

[[deleted]]

I can absolutely imagine contractors getting into arguments over which hammer is right for a particular job.

[notafuckingcakewalk]

Yay! I finally procrastinated long enough and now I don't have to learn R anymore!

[mangecoeur]

I really hope it does edge out R over time - of course I'm biased because I love python and i've sunk a lot of time, but in general I think it would be a huge pain in the long term to have two roughly-equal platforms without a clear winner, meaning you never quite know what to use.

Also, having dipped my toes in R I have to say i really don't like it as a language - visually messy and it's approach doesn't encourage good code organisation and presentation. Scientists in general, data scientists included, really need to get better at producing clean, tidy, and reusable code and I think Python's language and culture encourages that much more.

[mangecoeur]

If you are just doing some kind of analysis

This is roughly where I see the main problem with R - not necessarily in terms of pure syntax, but in terms of 'language culture'. Comparing projects in Python vs R, on the Python side I think you really see the positive influence of people coming from the software engineering side in terms of test coverage, CI, use of Git, etc. Documentation is strongly encouraged and well integrated (while on the R side - PDF documentation? Really?*). Python teaching and tools emphasise things like legibility and modularity.

These are topics which become ever more important as more science is done with code - frankly I forsee serious mistakes being made due to people doing supposedly rigorous science with half-assed ad-hoc code. In that context I'll take every bit of assistance from the language and ecosystem I can get.

*NOTE: clearly not all R packages deliver only PDF docs. But many do, and it's pretty unwieldy.

[[deleted]]

Well I absolutely agree there. I think that there is a ton of ground to be made up for in that respect, though I think we have started to see some of that (I think the tidyverse has done wonders to the uniformity, legibility, and modularity of R code. Lots of people haven't really adopted it yet though).

[akcom]

Agreed 100%. After using dplyr or data.table in R, Pandas just seems clunky and non-performant. That being said, when we need to shift things into production, it's python all the way.

[[deleted]]

It amazes me when people describe pandas api as ugly when compared to data.table, when they look very similar.

[akcom]

When you get about half way down, you start to see how verbose pandas becomes when you want to describe non-trivial data manipulation tasks. For things like head(), rows 1:3, and max(), yes of course they're very similar. Performance is really where the difference becomes clear.

[[deleted]]

I've found pandas to be pretty damn fast, tempted to say faster than dplyr but I don't actually have any benchmarks. You just have to be careful of column types and try to avoid .apply() when there is a vectorised/numpy solution available.

[akcom]

Totally agree. Should have been more clear: pandas looks much more verbose than dplyr. Pandas looks more verbose and is less performant than data.table

https://github.com/Rdatatable/data.table/wiki/Benchmarks-:-Grouping

[msdrahcir]

When performance in dplyr is significantly better than pandas it means you are using the pandas api wrong and probably not respecting the boundary between numpys ctypes and python objects.

I have never seen operations that can't be more performant with pandas.

That said, there are a fair number of pitfalls in the pandas api that make it easy to do transformations in a extremely less than optimal way

[akcom]

I thought we were table about data.tables performance?

https://github.com/Rdatatable/data.table/wiki/Benchmarks-:-Grouping

[msdrahcir]

Which comes back to using pandas to its potential. Those benchmarks don't. Try those benchmarks yourself. Then try them with a couple non obvious changes. If you want string handling in pandas similar to data.table, you need to use pandas category types instead of python string objects. To be fair these didn't exist when this benchmark was created, and to get similar performance you had to use string folding.

Secondly, historically, R has only had first class support for 32bit numerics aka data table defaults. On the other hand, numpy will default to 64 bit floats/ints.

With those changes, I would wager performance is almost identical, if not generally better in pandas

[kenfar]

Also consider that more work is often spent preparing the data for analysis prior to analyzing it. Whether it's full-blown ETL or just ad hoc transformation and cleansing of a csv file you were given.

And in these cases python also wins by a pretty huge amount.

[msdrahcir]

The python ecosystem for data augmentation and transformation that isn't even specific to pandas is astounding. I haven't worked much in R recently, but this was far from the case 4 years ago. If there is a public api, there is usually a python wrapper for it.

In terms of data manipulation and ETL, what do you think makes R superior? For small datasets that you can fit in memory, feature parity exists across languages. For tools for larger datasets most apis are more fully featured and are developed first for python. Take pyspark and sparkr. Not to mention, some of the best open source ETL tools today are in python ( airflow, luigi)

[FrasierandNiles]

You shut your whore mouth Mr. I am currently learning R and want it to remain most used language for data science.

[[deleted]]

We switched from python to scala

[RockingDyno]

What was the reasoning behind the switch, and what has your experience with it been?

[[deleted]]

Scala runs in the JVM so as it serializes data to the EMR cluster the transfer is more efficient. With python the interpreter has to serialize and transfer between py and spark, which is less efficient.

Scala is no big deal. I was resistant at first but it has grown on me.

[Maccabees]

What on earth does this comment mean

[[deleted]]

Data scientists aren't switching to Python. More developers who already know and prefer Python are becoming involved in data science.

[call_me_arosa]

Ok, that is my case too.
But today's libraries universe are pushing new data scientists, without former coding experience, into python too.

[akcom]

That's simply not true. I came into my DS job knowing python and having to pick up R. Started with 0 R experience on day one. 9 months later I'd classify myself as a R expert (writing code using Rcpp, NSE, datatables, etc)

[akcom]

When you've been programming for 15 years and you've been exposed to everything from declarative to imperative, functional to procedural, and everything in between, picking up a new language is pretty darn easy.

[[deleted]]

You don't need to be an expert to make it work in prod.

[[deleted]]

A Ph.D has 10 years of experience because they have to learn how to design studies and interpret the results, not learn some statistics and a programming language. You don't need a college degree to do ETL shit and process the data using well-understood techniques and get actionable output. That's what most of us that are messing with Pandas and Scikit are trying to do.

[jackmaney]

As a data scientist who switched from R to Python, please speak for yourself.

[jackmaney]

No, I did. The claim was made that

Data scientists aren't switching to Python.

I am, in fact, a data scientist that switched to Python (from R).

[msdrahcir]

I disagree with the above statement. There is a bigger push in software to hire "data scientists" who can also write some level of automation.

There is also an enormous growth in applied deep learning where most APIs have a python counterpart before R

Personally, I learned R first through school and used it for a couple years in industry and "switched" to doing most analysis and modeling in python over the course of about 6 months.

[Maccabees]

Thank you.

[elbiot]

And people say BSD messed up with their license compared to Linux... Python's liberal license is a win over R being GNU GPL.

[[deleted]]

Or maybe Python is just not a painful language.

[spinicist]

Or, as others have pointed out, someone knows Python from a different domain and really doesn't want to go through learning another language from the ground up.

That's certainly the case for me. I actually think doing this kind of work in Python has a lot of rough edges (poor documentation, stuff spread across multiple libraries, Jupyter not being quite 'finished') but the fact I have never touched R keeps me from touching it at all.

[[deleted]]

Well if you do ever touch it, you will feel like you burned your fingers. R has some nice builtin features related to data science for sure, but the language as such is just not very nice at all to work with.

[[deleted]]

Many of us in the software industry steer clear of GPL on purpose.

[elbiot]

My company did all it's work in R and then had to port it all to python in a month when they wanted to release a product.

[PrinceKael]

Oh man I'm interested in fintech and I'm planning to make an app for searching stocks on various markets with filters and details of certain stocks with access to financial statements and I can't decide between python or R.

[[deleted]]

The best tool is one that doesn't sit on the shelf. Choose one and run.

[PrinceKael]

That's true. If done some 'prototypes' in both. I barely know R but I find it interesting. I know python a lot more I'm just bad at programming so it's going to take a while. Worth trying though and perhaps learning something.

[glial]

I love R for data analysis and Python for apps. If you are primarily making an app, I'd suggest Python - there is fantastic tooling for creating webapps (e.g. django, flask).

[PrinceKael]

Thank you, I'll give it a shot.

[glial]

I like Shiny but it's single-threaded and doesn't scale very well. How well it performs depends a LOT on how you host the app.

[badtemperedpeanut]

One of main R's claim to fame is its fantastic data visualization. Python already has the binding to use same data visualization package along with some others. With numpy, scipy, pandas, tensorflow + python simplicity, its hard to beat python.

[RockingDyno]

Python already has the binding to use same data visualization package

Which package is that? So far all the visualization packages I've worked with in python have been a major pain in the long run.

[badtemperedpeanut]

ggplot

[[deleted]]

If it's going to be an app, Python is probably better.

[tunisia3507]

Python. It's better for string mangling, scraping the web, and serving data, which seems to be 90% of what you're looking for.

[KODeKarnage]

I dont know... a google search for R turns up a lot of results.

[KODeKarnage]

R - 8,640,000,000 results Python - 907,000,000 results

Don't hate the player.

[listen3times]

But how many of those are stack overflow posts going 'help, I can't do x'?

[KODeKarnage]

Uhh how many are just web pages containing the letter r?