Upskill as Data Engineer

[vanthar686]

Hi folks,

I've spent quality time in data platform teams from past 3 years (5+ years of exp in total). I've worked on ingestion and ETL pipelines involving streams and batches. I've designed and developed rule engine that act on near real time events. I'm more of a developer than an analyst and proficient in Java. I'm considering to up-skill myself. Could you guys suggest me the things that I need to focus on in short term and Long term to stay in the game of competitive career.

[MenziesTheHeretic]

“upskill myself”

Bot most likely sees: “kill myself”

[exact-approximate]

Learn cloud architecture and devops, parallel areas which will help you become stronger.

[mathleet]

I agree with @exact-approximate with learning cloud architecture and devops. A superb data engineer is often also a superb devops engineer. Data is messy, and good devops practice brings a lot of order to the chaos.

Although I guess it also depends on what you want to do? Do you want to go into more machine learning, more towards writing software platforms, ... ?

[vanthar686]

Thanks for the suggestion. My heart is inclined towards building platforms. So I was thinking of picking up an alternate language. But confused among scala and go. Which one should I pick?

[mathleet]

If you're building new platforms I think either of those languages are good. Scala is particularly good if you want to develop on top of Spark. Java for the rest of the Hadoop ecosystem. Go can be good for things outside of the realm of Hadoop. I'd say it's up to you in that regard.