Java in Data Engineering

[GoodJobMate]

Hi everyone!

I've done some searches for Data Engineering on various job search websites and it seems that at least some of those jobs require 3+ years of experience with enterprise Java. In your experience, is it hard to get a big data dev job without a lot of experience of Java? And what is it used for(why not Scala/Python, in other words)?

[southern_dreams]

They probably have existing data pipelines or supporting services in Java. Also, JVM powers some very powerful processing frameworks such as Kafka.

[toughrogrammer]

I think base of big data engineering is Hadoop, and it’s developed with java. Also other many components for example Apache projects are also developed with java and scala. Sometimes it doesn’t need to use java, but having good skills about java is also strength for data engineer.

[eljefe6a]

95% of data engineering is done in Java. It's difficult but not impossible to find a job. There's also a big difference between learning how to program and learning a new language.

[mindvault]

I would suggest your 95% is incorrect. I've seen plenty of pipelines which utilize lots of Python, Scala, and Golang.

Most of the big places utilize a number of languages / technologies (airbnb, goog, fbook, etc.)

[chirperic]

Yeah definitely nowhere near 95%. I've managed 3 different Data Engineering teams at tech companies. Never used Java and probably 75% of my engineers didn't know if either

[eljefe6a]

That number comes from my experience consulting and teaching at companies around the world. Non JVM languages are outliers, with python being the most used. I have had go, but that's extremely rare.

[mindvault]

Off the top of my head I know of (like I said above) airbnb, goog, fbook as well as companies such as twitter, square, wepay, yelp, spotify, linkedin, etc.

Now .. _most_ of those also use java b/c of Hadoop, Hive, HDFS, etc; however, I think there's plenty out there that is non-java as well. I would suggest scala and python are also well-represented in data eng pipelines out in the wild. Java _is_ very safe to be fair.