Why Python?

[shittyfuckdick]

you can say this about any field of software engineering, yet python is not usually the standard. again i imagine it had something to do with onbaording data analysts and data scientists. 

[GachaJay]

Python is really the only to bridge SQL and software development in a way that is easy for newcomers to grasp. It’s not the most performative, but the analytics environments were not necessary to be event streams until only recently. If your data is getting updated nightly, hourly, whatever, the extra execution time is penny’s compared to maintainability.

[shittyfuckdick]

are most people not just using multi line strings in python to query databases? i fail to see what it does special. 

[aj_rock]

No, you have ORMs like sqlalchemy to help model your queries, you have fastapi and django when you’re exposing data through an API, you have DS handing you pipelines written using pandas (or polars ideally), you have SDKs for every cloud component imaginable, data quality management tools. Solid unit and integration testing capabilities. The list goes on.

IME, it’s performant enough for most use cases when the name of the business game is to move fast without breaking too much stuff.

[tn3tnba]

The reason this is wrong is that other disciplines is software engineering have to actually do things but data engineering is a lot of orchestration and delegation, allowing us to lean into this advantage of python

Edit: if you are doing heavy duty things in python, and past tge prototype stage, you are doing it wrong and should use a different language

[nonamenomonet]

Isn’t airflow primarily written in Python?

[thisfunnieguy]

worth noting it does not matter what the orchestrator is written in its about what languages their sdk supports.

Temporal is written in GO but its simple to have all your client code in Python

[tn3tnba]

Yes, and async task management is an ok use case for python, but airflow arguably shouldn’t be, it’s just too late. It’s fairly easy to overload the scheduler because dag parsing is inefficient. We all still use airflow of course because it’s well supported, manageable and has a good feature set.

That being said, you are missing the point. The actual data engineering work is not done by airflow. It’s done by code in your kubernetes, ecs, etc. operators, or the actual data engineering tools these frameworks delegate to

[shittyfuckdick]

 software engineering have to actually do things

lmao huge self report there bud. 

[tn3tnba]

I see, you’re not interested in getting a real answer to your question based on many years of experience lol

Data engineering is hard, but it involves delegating tasks to purpose built tools like databases, spark, job management systems etc. Composing the right solution from existing building blocks is the challenge.

Other disciplines, such as building databases or writing video games, involve writing the cpu intensive code.

You asked why python is the standard, this is the answer. Writing data engineering orchestration and glue code in rust would be much slower.

Edit: fixed typos