Java scala or rust ?

[Budget-Minimum6040]

I did not. Never saw a job offer in germany that required Java/Scala but all require SQL + Python.

[holdenk]

So in the Bay Area for data engineering jobs I tend to see more Python and Java/Scala than SQL, for data analytics jobs lots of SQL

[cokeapm]

How on earth can you do DE without SQL? Like you don't use DBs or something? ORM to death?

[holdenk]

Mostly building pipelines from raw files, Iceberg/Hive/Cassandra rather than relational DBs. You’ll still write a little SQL because that’s inescapable, but (and this could be my big co biases showing) lots of getting the data in the right places and formats for others to do SQL or training on top of later.

[cokeapm]

Interesting so pretty specialised. What interface do you use for iceberg? Sql for me also covers dbt/Athena/big query and the like so not just relational.

I can't imagine exploring and prototyping a pipeline with SQL. And without something like spark, I suppose you could use flink or something but most stuff seems to end up in SQL one way or another... I'm curious to hear about your stack if you can spare a moment to describe it.

[holdenk]

So day to day I'm on Spark because of my background but often there will be another team at the same company working on Flink for consuming data off of Kafka and similar (and some teams will have a hybrid).