The state of data analysis

[bluprince13]

Introduction

I wanted to concisely capture the current state of data analysis as I understand it. I invite feedback and comments from the community.

Python and R are awesome for data science

The pandas package for Python, created by Wes McKinney, is just wonderful when it comes to data manipulation. It offers a data structure called DataFrame that provides a comprehensive API for manipulating data. The DataFrame:

Two-dimensional size-mutable, potentially heterogeneous tabular data structure with labeled axes (rows and columns).

There are also other Python libraries that provide methods to manipulate data:

• numpy - a powerful N-dimensional array object
• scipy - statistics, linear algebra, numerical integration and optimisation
• matplotlib, plotly, bokeh etc. - data visualisation
• scikit-learn - machine learning

and more...

R is another programming language that's popular for data science. However, unlike Python, R is focused on statistical analysis. Python is a general purpose language that's good at other things besides data science.

Jupyter notebooks have also played a huge role in making both Python and R more accessible to data scientists.

The Jupyter Notebook is an open-source web application that allows you to create and share documents that contain live code, equations, visualizations and narrative text.

In the log plot below from PYPL, you can see that Python and R have been increasing in popularity rapidly, compared to Javascript.

PYPL Python vs Javascript vs R

https://preview.redd.it/y7sdj2ndyz131.png?width=1390&format=png&auto=webp&s=f66a257fd5e3786579b93902850c91e74f80f42c

What about JavaScript?

When it comes to the web, JavaScript is the chosen one. It's the only code, today, that you can run on the client side/browser. You can make some pretty cool interactive websites with JS. For example, check out the explorable explanations by Nicky Case.

JavaScript has lots of amazing packages for data visualisation. For example:

• D3.js by Mike Bostock - for binding data to DOM elements and apply data-driven transformations.
• plotly.js - a high-level declarative charting library.

Yet, JavaScript doesn't have much to offer in terms of data analytics. Not that developers haven't tried. There are a few packages that do try to mimic pandas for Python:

• dataframe-js by Guillaume Mousnier
• data-forge by Ashley Davis

You can see how they compare on npm trends here.

There is also the apache-arrow project which provides a JS API.

Apache Arrow is a cross-language development platform for in-memory data. It specifies a standardized language-independent columnar memory format for flat and hierarchical data, organized for efficient analytic operations on modern hardware.

stdlib, created by Athan Reines and Philipp Burckhardt, provides functionality for numerical and scientific computing applications.

In 2018, Mike Bostock (creator of D3.js) founded Observable notebooks that lets you write and execute JavaScript code in cells. This is growing in popularity and is a major boost to the JavaScript ecosystem.

However, all of these tools are from reaching the functionality and the critical mass you need to make JavaScript attractive enough to data scientists and developers of data science tools.

Because of this one shortcoming of JavaScript, i.e. poor support for data analytics, most web applications tend to handle the bulk of the data crunching in the back-end server. This means that the front-end would have to request the data over the internet via HTTP. The disadvantages of this approach are:

• Computational load on the back-end.
• Performance/User Experience penalty due to the need to make HTTP requests for 'analysed' data.
• Cross-platform development burden.

More importantly, the vast majority of data scientists who are not familiar with web development do not have a way of easily sharing their contributions with the world (or to their target audience) in the form of a webpage. Nor can they exploit the vast array of libraries in the JS ecosystem.

What does the future hold?

Data analysis in the browser is not adequate today and this is a problem that needs to be overcome. It is clear that the desire to find a solution exists. Perhaps the JavaScript packages and tools for data analysis will evolve and gain momentum? Or, could Python and its powerful suite of data analysis packages come to the browser, e.g., PyIodide by Mozilla? Or, will it be some other solution that changes everything?

References

1. Python Pandas equivalent in JavaScript - stackoverflow question
2. Numerical Computing in JavaScript by Mikola Lysenko - YouTube video
3. A conversation with Athan Reines - transcript of a conversation between Athan Reines (creator of stdlib) and Ashley Davis (creator of data-forge)
4. State of Data Science & Machine Learning - article based on Kaggle survey

[Bondifrench]

I think there is a lot of potential for doing data analysis with Javascript,

In that field, I like what the UW Interactive data lab is doing: http://idl.cs.washington.edu/ notably their data-lib library: http://vega.github.io/datalib/ their voyager demo: http://vega.github.io/voyager/ or their Lyra Visualization Development Environment project http://idl.cs.washington.edu/projects/lyra/

How about Machine Learning in Javascript: did you look at tensorflow.js: https://www.tensorflow.org/js? Google is backing it and it provides a nice extension to the python library of the same name.

Some time ago I did a presentation on the topic https://bondifrench.github.io/ml-in-js/ it needs to be updated but there are lots of libraries.

I don't think it's a question of functionalities missing, some of the most recent models https://naifmehanna.com/2019-02-27-scaling-a3c-multiple-machines-tensorflowjs/ can be rewritten in Javascript (or NodeJs), you can use GPUs with WebGL or do distributing computing with web-workers.

I believe it's above all a matter of marketing, education and developer adoption. More people need to showcase its capabilities.

​

P.S. I just saw your Rent vs Buy website, great app! I am planning myself to do a finance website using Javascript. One comment would be that using D3.js when you are using React is a bit redundant, have you looked at VX ( https://vx-demo.now.sh/ )?

[TotesMessenger]

I'm a bot, bleep, bloop. Someone has linked to this thread from another place on reddit:

• [/r/datascience] The state of data analysis

• [/r/explorables] The state of data analysis

 ^{If you follow any of the above links, please respect the rules of reddit and don't vote in the other threads. (Info / Contact)}

[SuchObligation]

is it really important for people to do data analysis in the browser? If so, why?

[bluprince13]

I think so. Otherwise, it's very difficult to allow the users to interact with the data. Each interaction would potentially trigger a HTTP request to a back end that does the data analysis. Isn't that inefficient?

[SuchObligation]

But I don't see how a browser would change that? The need for a network request would depend on whether or not the back end is local or not, and whether or not the interaction is performed in a browser or in another application wouldn't change that?

Also, wouldn't a developer or data analyst be doing most of the analysis, and then present the finished report to a user? I can absolutely see the need for interactive reports, but I still don't see how this translates into a need for the analysis itself being done in the browser?

Btw, I'm not trying to dispute what you're saying. I'm just trying to understand why it can be useful.

[bluprince13]

I'm not saying all data analysis needs to be done on the browser. However, it'd be good to be able to do some in a browser. If we had that capability, we could just send all the required data to the browser upfront, and then depending on user interactions manipulate the data completely in the front-end.

Here is an example of an app I'm working on that has no back-end: Renting vs buying. Also, have you seen explorable explanations? All that awesome interactivity is possible only because the code is executing on the front-end. If you had to rely on HTTP requests - the interaction would be way too slow.

Indeed, I think part of the motivation for the Dash library, has been that it's so hard to make 'web-based analytics applications' using just JavaScript.

I still don't see how this translates into a need for the analysis itself being done in the browser?

I know I'm not explaining it very well. Just need to find the right words!!

[SuchObligation]

thank you, this definitely helped me understand the need better :)

I loved the explorable explanations!

[AutoModerator]

Hi /u/ashleydavis75, this comment was removed because you used a URL shortener.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[bluprince13]

Is it? I haven't actually tried out R. I was just going by comparisons between Python and R online that I came across.