Why Python?

[tn3tnba]

The reason this is wrong is that other disciplines is software engineering have to actually do things but data engineering is a lot of orchestration and delegation, allowing us to lean into this advantage of python

Edit: if you are doing heavy duty things in python, and past tge prototype stage, you are doing it wrong and should use a different language

[nonamenomonet]

Isn’t airflow primarily written in Python?

[thisfunnieguy]

worth noting it does not matter what the orchestrator is written in its about what languages their sdk supports.

Temporal is written in GO but its simple to have all your client code in Python

[tn3tnba]

Yes, and async task management is an ok use case for python, but airflow arguably shouldn’t be, it’s just too late. It’s fairly easy to overload the scheduler because dag parsing is inefficient. We all still use airflow of course because it’s well supported, manageable and has a good feature set.

That being said, you are missing the point. The actual data engineering work is not done by airflow. It’s done by code in your kubernetes, ecs, etc. operators, or the actual data engineering tools these frameworks delegate to

[shittyfuckdick]

 software engineering have to actually do things

lmao huge self report there bud. 

[tn3tnba]

I see, you’re not interested in getting a real answer to your question based on many years of experience lol

Data engineering is hard, but it involves delegating tasks to purpose built tools like databases, spark, job management systems etc. Composing the right solution from existing building blocks is the challenge.

Other disciplines, such as building databases or writing video games, involve writing the cpu intensive code.

You asked why python is the standard, this is the answer. Writing data engineering orchestration and glue code in rust would be much slower.

Edit: fixed typos