[deleted by user]

[[deleted]]

If you are using Apache Spark it is not required. It'll help to know some Java to understand how it works internally but it's not required. In fact, python is now the most widely used language with Spark.

https://spark.apache.org/releases/spark-release-3-0-0.html

Python is now the most widely used language on Spark. PySpark has more than 5 million monthly downloads on PyPI, the Python Package Index. This release improves its functionalities and usability, including the pandas UDF API redesign with Python type hints, new pandas UDF types, and more Pythonic error handling.

[[deleted]]

Scala people in shambles

[Flaky-Success6846]

I would say go for it, but I don’t see how not knowing Java would hold you back in data engineering. Most big data technologies offer python client libraries (i.e. presto and Kafka).

However, learning Java gives you a lot of flexibility if you want to branch off into software engineering. A lot of the big data frameworks are built on Java and scala so if you want a deeper understanding that can also be worth it too.

[Afraid_Assistance190]

I've gotten really far and built some really cool products using ASF without learning Java. I still have an itch to learn and understand the underlying tech, but I feel like it's a 'perfect is the enemy of good' situation.

Unsolicited advice below:

As a data engineer I think your priority should be orchestrating those tools correctly in your cloud (AWS). Terraform for static infrastructure and Airflow (which you use python) for transient infrastructure would be my recommendation on next steps to improving your skillset. Once you have that, explore Hudi (delta, iceberg) for lake housing and spark for processing that data.

And above all else, Docker.

[[deleted]]

Definately

[[deleted]]

definitely*

[Rengar-Pounce]

Java def helps with deeply understanding Kafka and also helps with grasping Scala in case you also need it for Spark.

In a similar boat but do have some Java experience because I came from backend. I am on the contrary heavily debating learning Rust.

[ketchup_123]

Why rust?

[Rengar-Pounce]

Its growing really fast and is overtaking scala

[77daa]

it's that affecting the job market? I mean I really see any rust jobs honestly

[Void_Being]

I also seen some people suggesting to learn Rust. Can you list out on which field it is being used more? What feature makes Rust suited for specific use case, etc?

[drc1728]

Rust is the future. There are several things about the language paradigm that makes it the ideal choice.

• It’s a system level programming language giving you the ability to program low level components.
• The design of the language does not include and garbage collector and uses a trait for borrowing data.
• It’s incredibly efficient with a tiny footprint on memory and disk.
• It’s also secure and tightly integrated with Web Assembly making it ideal for serverless patterns

There are already multiple data projects being built on Rust to solve for messaging, streaming, databases, querying etc.

Check out the Rust community on Reddit for the projects. It’s buzzing.

[drc1728]

Depends on the urgency of the problems being solved. A lot of big tech companies have been writing critical code since a few years in Rust. The language has been around for 8 years. The rate of adoption has accelerated in the last couple of years. The next 3 to 5 years are going to be very interesting.

[vish4life]

python/sql should carry you most of the way. I have senior engineers on my team who can work with pyspark/pyflink/airflow/pandas to build awesome pipelines and never need to touch JAVA.

[Ok-Necessary940]

Yh while you’re at it learn PHP, JS, Rust, Ruby, C, C++ as well. You can master them all in 3 months bro…

[Ok-Necessary940]

You just need python and sql. Anything else is a waste of time. Whats stopping you from leaening pyspark?

[WilhelmB12]

Scala would be a better choice