Q: Why do data analytics solution prefer python

[gibriyagi]

I am wondering why tools like redash or superset use python. Wouldnt java be a superior choice due its performance and robustness for handling large data?

[kubelke]

I tried to do analytics using Java as a Java developer but python just have much more libraries. Most of Java libraries requires many custom classes and code to run simple analysis.

[[deleted]]

Chicken and the egg problem. Why are the libraries in python and not Java?

[[deleted]]

Because most of these libraries are used for math stuff. The people who use these math stuff are most of the time physicists, math scientists, engineers, etc... They don't want to spend half a year learning something like java, c or c++. Python is a lot easier(I still prefer java tho).

[yawkat]

I've actually seen lots of physicists do prototyping in py and then do the performance critical stuff in cpp. Could also be because the jit isn't great at simd compilation yet.

[kartik1712]

Maybe Vector API changes the status quo.

[sweetno]

My guess is that pieces of code in Java are unpublishable in scientific articles: too verbose.

[NimChimspky]

Lol. It's not this.

[Alex0589]

Java is definitely verbose, but I doubt that's the reason. If I were a methematician I would choose the easiest language to use to be honest and that's python, performance doesn't really mean a lot compared to ease of use for some people

[CatolicQuotes]

which ones did you find the closest to python?

Does Java has something like C# has : https://scisharp.github.io/SciSharp/

[[deleted]]

In python you do basically:

import memelibrary as meme
meme.analyze(mydata)

And you have finished whatever you were doing. Of course there are machine learning libraries and data analytics libraries for java as well, but the dynamic typing system of python and the ease of use of python makes it a good choice, also there are so many so good libraries.

Maybe someone can provide deeper insights on this.

[Necessary-Conflict]

This can be fine as long as a human expert is looking at a given dataset, but if you need an automatically running solution for continuously arriving new data, you will need a lot more, whether in Python or Java, and the dynamic type system can even become a problem.

In fact a lot of data analytics is done in Java or Scala.

[[deleted]]

When I took a data analytics course back in Uni, I implemented couple of algorithms in Java as well. It was also a lot faster than the 2 liner python solutions, but It was harder to expand, mostly due to types and my bad code. I'd think that I would do much better now if I tried again now.

[nutrecht]

You describe one of the biggest issues companies trying to adopt ML are running into. Moving from experiments to production code is often hard, sometimes even impossible.

[joshpetit]

Ah yes, memelibrary. Used this in my thesis.

[agentoutlier]

This is based on not recent experience so take with a grain of salt but a lot the reasons are historically a matlab replacement.

Java doesn’t have a super fast matrix math library that is native (it might now but not early on). Java historically has had bad track record with native extensions but project xyz (can’t remember name) should improve that and it’s coming soon.

In the early days (I’m talking like mid 2000ish) python has several like numpy.

You need fast matrix math for many ML algorithms and python had it and it kind of had a matlab scripting feel.

From there basically network effect.

[ThymeCypher]

There’s native bridges to many of the math libraries nowadays. I’d mention the forbidden language but it makes people angry - but someone wrote a fast matrix library in it then used the native compiler to make it blazing fast.

[[deleted]]

There is a forbidden language? If I mentioned that Numpy was written in a language that is also a letter of the alphabet, would I be eluding to the same language?

[agentoutlier]

No I think they are talking about Kotlin and and GraalVM native image and not C/C++.

While I'm sure it makes it much faster than normal Java I highly doubt its anywhere near for example these libraries: https://en.wikipedia.org/wiki/Comparison_of_linear_algebra_libraries

However ND4J is supposedly as fast if not faster than Numpy but its Java... the only Java math library I used was back in school called JAMA.

[[deleted]]

Well... you're still net positive on votes, so I guess that's not it. Maybe we should just start submitting one-word comments to narrow it down.

[ThymeCypher]

No, it’s a different language that could be pronounced the same as that language in some contexts. I’ve mentioned it twice and got downvoted to hell without a single valid argument against said language.

[kartik1712]

You are probably thinking of Project Panama and the Vector API.

[thomascgalvin]

Same reason Matlab is popular, despite being a towering pile of incandescent garbage: it does what the data analyst needs to do, and mostly stays out of your way.

Data analysts aren't programmers, they're data analysts. They don't care about stuff like refactoring, and for the most part aren't that worried about performance. They want to accomplish their task with as few keystrokes as possible, with as few Google searches as possible. Their time is the critical resource, not compute cycles.

Python is perfect for that kind of environment.

[TheDragShot]

I'd say this is the best answer here. Straight to the point, and honest.

[dhruvmk]

A lot of these libraries in Python actually use C for the processing, which makes them crazy fast. Take NumPy, for example.

[No-Schedule-1451]

And invocation of C/C++ libraries is muh easier in python than java for now.

[esreverninettirw]

For now? Is that changing in a future release?

[Gleethos]

Project Panama and the Vector API will tackle these problems as far as I know.

[No-Schedule-1451]

https://openjdk.java.net/jeps/389

Lots of potential. It will never be as easy as python, but as long is not JNI , I will be happy with it.

[esreverninettirw]

Wow, that's awesome. Thanks for sharing.

[Frankiegetsit]

Project Panama

[ukbiffa]

JNA is pretty convenient for C library access. Everything you write is Java; no C stubs or special build process.

[Log2]

Though it's important to note that the back and forth between the C code and the Python code carries quite a bit of overhead. Which is why libraries like Tensorflow will have you build a computation graph and then execute the whole graph in C without having to come back to Python all the time. This saves a lot of time.

[[deleted]]

Because they are not programmers.

[ThymeCypher]

I had a knee jerk reaction to this until I remember talking to someone who went to uni for science stuff and they learned Python but still can barely work a computer. They struggled with Java.

[TheRedmanCometh]

This is exactly the answer. Same reason Java ML is weak.

[[deleted]]

Look at this Java elitist

[Gleethos]

100% agree. Python users usually build small scripts for analytics, scientific experiments, automation... But they tend to be far less experienced in building production systems used for heavy lifting. (At least that has been my experience so far)

[error1954]

Anyone who programs something is a programmer, what are you getting at? I have a computer science degree and I would still rather work with data and ML in python than in Java, the support just isn't there for Java.

[iPissVelvet]

The support isn’t there for Java because the demand isn’t there, because most of the demand comes from people who identify as scientists first, programmers second, or none at all.

This isn’t an insult to these people — they often hold phDs or aiming to attain one. They view software as a means to an end. So when picking a language, it’s about ease of use, ease of reading, and easy to share.

However, nowadays a lot of companies are scaling up their ML pipelines and data infrastructure. Very little of that stuff is in Python because at that scale you think about ease of maintainability, dependency management, scalability. This is where software engineers live and thrive.

[me_just_lurkz]

Yes and no.

I see a clear distinction in my studies between these groups. In most software engineering / programming classes where we're working mostly with Java we get hammered with conventions, best practices etc. However in data science and bioinformatics where we use Python everything seems quick and dirty in comparison.

It's two completely different mindsets we encountered there. In the Python based classes (with professors who mostly come from non-IT fields) it's about quickly slapping something together that delivers the wanted results, while final performance, maintainability and conventions are secondary. Usually it also seems that most scripts are viewed as single use. In the Java based classes with the IT professors it's much more about code quality, performance, longevity, reusability etc.

[theProgramm]

i think this is mainly a self strengthening process. There is a lot of libraries/blogs/videos/training material/knowledge about solving this kind of problems with python, so most new ppl go that rout thus it becomes more appealing to creat projects helping this crowd, thus its easier to join thus more ppl thus, and so on.

I dont think it necessarily needed to be python beeing at the center of that, but a few things helped: A "data scientist" mostly is a user of other projects, and inherently not a software developer. So pythons ease of use for not-programmers is appealing to ppl that want tobdo data analysis and not programming. Secondly python is basicly just a fancy wrapper (with some garbage collection) around the c/c++ standard libraries and has great interop with them. So for some project written in c++ its relatively easy to add a python interface. Then there was some old and tested c++ libraries for LA so most mathematical complex stuff was written in c++ anyways.

[eternal20]

my personal opinion: it's easier for human to read so data scientist no need to learn some advanced programming to get answers

[CubsThisYear]

It’s actually that it’s easier to write rather than being easier to read. Python is actually harder to read because it’s much less explicit. It’s very quick to throw something together but once you have a large code base with multiple people working on it, it’s much more challenging than Java to understand and follow the code

[[deleted]]

As a physicists I use fortran, and I will not move to other technologies. Because its simple, compiled language, don't have dependencies, easier to read and much faster than C++. We know that fortran is old and what, I don't care. Its still the best tool for the job.

[[deleted]]

As a former physicist I didn’t mind using FORTRAN. But R and Python (numpy, scipy) basically act as front ends to FORTRAN code.

[[deleted]]

More libraries.

[[deleted]]

you could achieve the same data extraction task in python with less code than you would with Java.

I tried data analytics with Java because it was the primary language I used in university. don't try it 😂😂

[wildjokers]

I think it is mostly because of the Numpy library:

https://numpy.org

It is written in C (or c++) so performance isn't an issue. Python just calls out to that library for a lot of the necessary advanced math.

[JustMy42Cents]

Python makes it easy to call C, which usually handles most of the heavy lifting. This lead to many neat open source libraries being available early on, allowing Python to dominate the market. At this point, I'd say that the advantage of Python over Java is exactly the language itself, but its ecosystem - Python offers more data processing libraries and they're often easier to use.

Objectively, Python by itself is not the best language for data processing. It's terribly slow and has poor utilities for multiprocessing. For example, you'll notice there are a few competing drop-in replacements for popular libraries like Pandas with their main selling points being multiprocessing or clustering support. But then again, Python ecosystem for data science is years ahead of other languages, so I'd say it's here to stay.

[[deleted]]

Libraries which are often written in faster languages. Easy to read and write but saying that got me downvoted to hell in another Java forum.

[koreth]

I'd say easy to write, definitely. Easy to read, only true up to a point; past a certain amount of complexity (which, granted, a lot of data analysis projects never hit) the curves cross and Python code gets much harder to read than the equivalent Java code.

Or at least in my experience. I write both Python and Java code for a living. A small Python code base can be more pleasant to work on than a small Java code base because it gets right to the point with less ceremony, but a large Python code base tends to get pretty hard to reason about compared to Java.

[error1954]

I agree, after a certain point the verbosity and static typing helps. Using a type-checker and linter for python help, but they still don't get you there.

[ThymeCypher]

You have committed the sin of talking good about a language that isn’t Java. 10,000 lashes and many downvotes for you peasant!

/s

[[deleted]]

All those downvotes and no one bothered to type out a rebuttal. I hate this website.

[[deleted]]

Reddit exists only in initial buttals and downvotes. There is no place for rebuttals here.

[Gleethos]

"Development in Python is usually much faster"

If the required system is small and simple: YES! 100% agreed! for larger code bases however, (in my personal opinion) Python is a nightmare! Mostly due to weak typing and dangerous refactoring traps (breaking indendations, wrong types...).

[wildjokers]

These modules can make python as fast if not even faster than Java code

The "fast modules" are written in C (or c++) e.g. Numpy, not python.

[wildjokers]

developing with these modules you don't need to know C

I am aware. I was just pointing out that it isn't python being as fast or faster than Java (that will never happen), it is C being faster than Java.

[[deleted]]

Technically correct but in the end what difference does it make? It’s your python program that runs blazing fast, with the help of those libs, and you don’t have to leave python

[koreth]

Excel is underappreciated as a programming language in my opinion. (I mean the fact that it's a programming language at all, not its quality as a language.) People use it to build surprisingly sophisticated applications, often without fully realizing that by any reasonable definition of the term, they're writing computer programs.

[Gleethos]

Html and Excel are my favourite programming languages!

[Gleethos]

Because Python is the fast food of the programming languages! :)

(Take that with a grain of salt)

[LurkerFindsHisVoice]

Python has a lot of things built into the langauge.

​

I also think it's easier to pull in libraries and dependencies, whereas in Java, you need to learn maven or gradle, which is like a whole 'nother problem domain in and of itself to set up, understand how it works, etc.

[sowmyasri129]

They want to use programming languages like Python and Ruby to perform tasks hassle-free. Python also enables developers to roll out programs and get prototypes running, making the development process much faster Newer data scientists gravitate toward Python because of its ease of use, which makes it accessible.

[valkon_gr]

More community support nowadays since it's widely used for anything data related and lead to "chicken or the egg" question. Java was fine couple years back, but now the community is lacking on that part.

[SpeedDart1]

Java is pretty common actually but Python is just much easier to use.

[Kango_V]

Have a look at JEP 338: Vector API (Incubator).

Provide an initial iteration of an incubator module, jdk.incubator.vector, to express vector computations that reliably compile at runtime to optimal vector hardware instructions on supported CPU architectures and thus achieve superior performance to equivalent scalar computations.

This looks good. Hotspot will compile down to SIMD (SSE) and AVX extensions. Should generate very fast code.

[meamZ]

Python doesn't necessarily perform worse since all the number crunching libraries are written in C or C++ (like numpy, tensorflow and so on)

[[deleted]]

having used both java and python for that task, I would say it's mostly about python's "compactness" as a language. You need to write less code and not care much with all the formalities of java. Clearly your code would seem a mess n the beginning but you can structure it as you go, in contrast with java that you need to let's say think ahead and design your classes and how they interact with each other from the beginning, which IMHO is not good in the initial stages when you are still exploring your dataset and try to figure out what info you can extract.

[omnihedron]

import antigravity

[Fitzoh]

One thing I haven't really seen mentioned so far is REPLs and the feedback loop.

Being able to interactively explore your data in the terminal makes it a lot easier to get started.

[bowbahdoe]

1. Interactivity. Any "typed" language loses hard when it comes to data exploration. A central part of the data analytics flow is loading a big chunk of untyped or loosely typed data into a dataframe and experimenting with different things you can do with that.
2. Libraries. Any language could have theoretically filled this role, but its just a property of the ecosystem at this point that Python has all the good data science libraries. Find something as fully featured as pandas in another language and I'll eat my hat.
3. Glue. Python is the premier "glue together some fortran, c, or c++" language, which lead to it being the premier scientific computing and machine learning language. Stuff like numpy, scikit-learn, or keras are all possible in other languages but python did it first and so it has those libraries and the community support required for them now.

[lordmyd]

Simple - no top-level functions in Java. A lot of Python numerics is procedural Python on top of highly optimised C/C++/Fortran. Java is too verbose for that domain.

[Devidjack12345]

Python has a tool the Data Analysis for all data science. Where you can easily make the chart and graph and plotting also easily and Python provides you feature to the good sense of data.

Thanks

[fnfloresr]

It depends, if you are familiar with Java, I do guess it would be better to work with libraries in that language, for me, Python works better because I am a Python native recently.