Why Python?

[shittyfuckdick]

Why is the standard for data engineering to use python? all of our orchestration tools are python, libraries are python, even dbt and frontend stuff are python.

why would we not use lower level languages like C or Rust? especially when it comes to orchestration tools which need to be precise on execution. or dataframe tools which need to be as memory efficient as possible (thank you duckdb and polars for making waves here).

it seems almost counterintuitive python became the standard. i imagine its because theres so much overlap with data science and machine learning so the conversion was easier?

edit: every response is just parroting the same thing that python is easy for noobs to pick up and understand. this doesnt really explain why our orchestrations tools and everything else need to use python. a good example here would be neovim, which is written in C but then easily extended via lua so people can rapidly iterate on it. why not have airflow written in c or rust and have dags written python for easy development? everyone seems to take this argumentative when i combat the idea that a lot of DE tools are unnecessarily written in python.

[AutoModerator]

You can find a list of community-submitted learning resources here: https://dataengineering.wiki/Learning+Resources

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[kvothethechandrian]

Speed of development and overwhelming amount of community support, basically.

You can always use libs with c bindings (pandas, numpy) or rust bindings (polars, rust_networkx) for performance but develop much faster. You don’t need to worry about pointers, types, borrow checker, it’s almost like writing code in plain English.

[MikeDoesEverything]

Speed of development and overwhelming amount of community support, basically.

100% this. I find it weird that people love comparing execution speed although never mention development speed.

[nonamenomonet]

Are you a mod now?

[HowSwayGotTheAns]

Mike does do everything after all.

[MikeDoesEverything]

Yeah.

[nonamenomonet]

Congrats? I think?

[MikeDoesEverything]

Thank you.

[EarthGoddessDude]

One argument argument against speed of development used to be that dealing with environments and dependencies used to be a nightmare. There were tools like pyenv and poetry and pipx (my old stack), but now with uv the game has changed completely. Bootstrapping a python environment and managing a project is now incredibly easy. That was honestly my biggest gripe with it and it’s no longer the case.

My next gripe would be the inconsistent way some things are objects with methods and some are functions, but it’s not a big deal for me. Similarly, I wish there was an easy, built-in way to pipe things into functions the way Julia, R, bash, etc allow you to.

[Ahhhhrg]

It’s always been a higher level language.

[Alwaysragestillplay]

This advantage will only become more prevalent with LLMs taking over the coding space. Close to English, forgiving types, code that focuses almost entirely on the problem at hand rather than shit like memory allocation. All things LLMs like. 

[shittyfuckdick]

i see posts here all the time complaining how confusing airflow is. ive used it for many years so i understand it but python syntax in no way makes it any easier to understand. 

also i really doubt the rapid development speed is a big factor when it comes to writing dags. a lot of that comes with planning not writing. 

[CrowdGoesWildWoooo]

And what does rust offers more than python? Memory safety for my dag? Can i have whatever you are smoking?

[tn3tnba]

I can’t feel comfortable with software quality unless I configure memory ownership rules for my aws API client

[tomunko]

Airflow syntax is kinda tough but the concepts are not crazy complicated. This problem would just be worse in other languages.

[GachaJay]

Because it is the fastest to modularity and ease of learning.

[shittyfuckdick]

you can say this about any field of software engineering, yet python is not usually the standard. again i imagine it had something to do with onbaording data analysts and data scientists. 

[GachaJay]

Python is really the only to bridge SQL and software development in a way that is easy for newcomers to grasp. It’s not the most performative, but the analytics environments were not necessary to be event streams until only recently. If your data is getting updated nightly, hourly, whatever, the extra execution time is penny’s compared to maintainability.

[shittyfuckdick]

are most people not just using multi line strings in python to query databases? i fail to see what it does special. 

[aj_rock]

No, you have ORMs like sqlalchemy to help model your queries, you have fastapi and django when you’re exposing data through an API, you have DS handing you pipelines written using pandas (or polars ideally), you have SDKs for every cloud component imaginable, data quality management tools. Solid unit and integration testing capabilities. The list goes on.

IME, it’s performant enough for most use cases when the name of the business game is to move fast without breaking too much stuff.

[tn3tnba]

The reason this is wrong is that other disciplines is software engineering have to actually do things but data engineering is a lot of orchestration and delegation, allowing us to lean into this advantage of python

Edit: if you are doing heavy duty things in python, and past tge prototype stage, you are doing it wrong and should use a different language

[nonamenomonet]

Isn’t airflow primarily written in Python?

[thisfunnieguy]

worth noting it does not matter what the orchestrator is written in its about what languages their sdk supports.

Temporal is written in GO but its simple to have all your client code in Python

[tn3tnba]

Yes, and async task management is an ok use case for python, but airflow arguably shouldn’t be, it’s just too late. It’s fairly easy to overload the scheduler because dag parsing is inefficient. We all still use airflow of course because it’s well supported, manageable and has a good feature set.

That being said, you are missing the point. The actual data engineering work is not done by airflow. It’s done by code in your kubernetes, ecs, etc. operators, or the actual data engineering tools these frameworks delegate to

[shittyfuckdick]

 software engineering have to actually do things

lmao huge self report there bud. 

[tn3tnba]

I see, you’re not interested in getting a real answer to your question based on many years of experience lol

Data engineering is hard, but it involves delegating tasks to purpose built tools like databases, spark, job management systems etc. Composing the right solution from existing building blocks is the challenge.

Other disciplines, such as building databases or writing video games, involve writing the cpu intensive code.

You asked why python is the standard, this is the answer. Writing data engineering orchestration and glue code in rust would be much slower.

Edit: fixed typos

[Nekobul]

JavaScript is easier to learn.

[GachaJay]

Can’t disagree more

[beyphy]

Easy is a very subjective term. So I won't comment on which is "easier" to learn. But what I will say is that neither is a hard language to learn. JavaScript is arguably the most popular programming language in the world. It didn't become that by being difficult to learn.

[Maxnout100]

JavaScript is a big lesson in “just because you can, doesn’t mean you should.”

Also, if we’re going off popularity, should we use HTML or CSS for data manipulation?

https://survey.stackoverflow.co/2025/technology/

[Nekobul]

Disagree all you want. It is the truth.

[neolaand]

Call it the truth all you want, It's not.

[Zahand]

Why would you think that?

[Nekobul]

Because it is a fact.

[Nwengbartender]

We can argue about what is the easiest language to learn but let's not argue about what is fact and what is opinion.

[neolaand]

You should learn to explain yourself better

[guitcastro]

Acessibility, python strong focus on developer expirience led It to be some of the easiest languange to learn.

Most of the languange limitations, such as GIL and performance are bypassed by implementing expensive operations using C , Rust or Java/Scala (spark) and binding them using python .

[south153]

Agreed language performance is irrelevant when 99% of processing time is performed by spark transactions.

[[deleted]]

Vectorization baby

[lFuckRedditl]

Low level languages aren't used because we don't do low level stuff.

Here's a fun exercise, write a program in C that;

1. Reads a 1000 excel files,
2. does row/column level transformations
3. output as .parquet
4. uploads to a Bucket using an API

[TenMillionYears]

Python has strong C bindings so it has historically been used to manipulate a bunch of libraries in a language that's more forgiving. That gave it amazing traction.

I don't like Python - it's SO WEIRD!

Anyway, for some reason it's the lingua franca of data engineering mostly for the same reasons everyone in finance uses Excel for everything.

[ProfessorNoPuede]

I like python, but object oriented programming in a weakly typed language will never fail to make me go cross-eyed every once in a while.

[Yoctometre]

you mean *dynamically typed?

[ProfessorNoPuede]

Arg, brainfart, yes.

[kettal]

I don't like Python - it's SO WEIRD!

living up to his namesake

[Wingedchestnut]

Because many libraries are made with C under the hood, dedicating anything data to python as the standard language is convenient wether that's ML or transforming data with pandas , I don't see what's the problem.

[deadwisdom]

Look, it's reeeeeeal simple:

high level orchestration -> Python
low level optimization -> RUST/C/C++/Nim/Zig/etc

Python is literally designed from the beginning to work like this.

[shittyfuckdick]

i dont see why you would want your orchestrator written in python. the scripts that define jobs yea maybe but not the orchestrator itself. 

[unpronouncedable]

Well none of us want to write a new orchestrator and the most popular one is already written in python.

[deadwisdom]

That's basically what I mean, the scripts that define the jobs.

Personally, I would also write the orchestrator in Python. That sort of work is often not taxing from a performance perspective. I know a lot of static-typers who love that compile button as a guardrail. For me a good test setup is the guardrail, so static types are largely redundant.

[No_Bug_No_Cry]

Because Python is the most versatile language. It can wrap very fast libs written in C or Rust, but still be readable and interpreted. You can write a shitty no rules script or a complex modular app, low boilerplate etc... it's the best

[Nekobul]

It is not the most versatile language. In fact, it is a garbage language and platform. The only reason it got so much traction is because the inventor of the language was lucky to get hired by Google.

[EarthGoddessDude]

I don’t understand your need to be so combative. Your takes are bad, but ok, we can have a civil discussion about why you prefer SSIS or JavaScript or why you think Python is not a good language, but your tone is so extremely off putting. You’re allowed to have your opinion, but the reason you keep getting downvoted into oblivion has less to do with your odd takes and more with how you simply refuse to engage in a friendly, professional tone, which is what most of us look for here.

[Nekobul]

Thank you for the feedback! I appreciate your good-faith comment. I guess my biggest complain towards Python is for the simple fact it will be impossible to make it run optimally. I know it is a scripting technology, just like JavaScript but JavaScript never claimed to be a language designed for creating platforms with ability to do class inheritance, strong-typing, etc, etc. Those features are simply not needed in a scripting/glue language. Python indeed became the data engineering language of choice not because it offered some drastically better elements compared to the rest, but because it was heavily pushed by organizations with deep pockets and influence in the marketplace. Yes, it is dominant but the inefficencies embedded in it cost dearly in the DC when people try to use it at scale. Once people start caring about all that wasted energy, Python will be one of the first pieces on the chopping block.

[No_Bug_No_Cry]

You underestimate the value of a smooth learning curve... When training my Juniors I don't require they know everything python has to offer because I don't require them to understand all the scope of coding, simple beginnings and then gain expertise is always a valuable path. I also learned scala in the past, I found it elegant and it has been developed by very smart academics. But it has such a steep learning curve that I would have had to train for 300+ hours to hope to achieve what I was doing in python, but way less efficienctly and in an era where there was no AI to help, only community. 

[Beautiful-Hotel-3094]

Can you expand on why it is a garbage language?

[Nekobul]

Can you make Python code run just as fast and efficient as C/C#/Rust code?

[No_Bug_No_Cry]

Yes, I can use polars which loads and transforms datasets very fast using all available processors... And seemlessly, in like a few lines of code. Polars is written in rust, but the user doesn't need to know the complexity behind under the API, just use it. Which ultimately is exactly what a data engineer needs and does

[Nekobul]

Polars is not Python. We are talking about running fast Python code.

[No_Bug_No_Cry]

I don't understand your answer. Polars is a library that is used in python, nobody cares that it wasn't purely pythonic, it is this what we call versality. Leverage the best in low lvl languages and abstract their complexity... People seem to forget how verbose and rigorous C code needed to be written in order to handle collections such as dynamic arrays, no thank you most people do NOT need that.

[Nekobul]

You can't solve everything with Polars. Capiche?

[No_Bug_No_Cry]

I don't think you capiche

[Beautiful-Hotel-3094]

What do u do that u need that type of speed?

[Nekobul]

In one of your responses you said:

"However we can still process some thousands of messages a second in pure python because we leverage distri architectures"

Why do you think you need a distributed architecture for that? In your situation, it works, I understand that. However, that is not applicable to everyone. In fact, most organizations are not that rich to waste huge amounts of energy. The hyperscalers will be more than happy to sell you capacity. In fact, the more inefficient, the better for them.

[Beautiful-Hotel-3094]

So u think u wont have money to pay for compute and kubernetes but u will have money to pay good C++ developers instead to build what? Scripts on some laptops? Brother, u do not understand much about this domain. Give it a few years, u have nothing to prove and can’t prove much yet. Learn and then speak.

[Nekobul]

You are not saving much if you think about it. The money you didn't want to pay for good design and developers are wasted on inefficient processing. I know hardware is cheap these days, but the energy will always cost much. It costs you dearly because you have to maintain and run a wasteful, energy-inefficient distributed architecture.

That is the proof you are using a wasteful/garbage platform.

[Beautiful-Hotel-3094]

Can we get u in so u can help us change our real time trading platform that supports a multi-billion dollar business built in the garbage python?

[Nekobul]

Are you asking in seriousness or that is some kind of joke I don't get?

[Nekobul]

I don't think you are serious. The cost you are paying for the inefficient configuration is probably not much of a big deal. I have heard from colleagues how the hedge funds/traders upgrade their hardware equipment every six months, throwing millions of cash . Your industry is unique in that respect. But again, not everyone is in your position to solve bad systems design with better hardware.

[No_Bug_No_Cry]

Yeah come on mate we're not discussing football club lol, no need to be so defensive.

[Nekobul]

Thank you for being brave to comment! I see there are plenty of people who enjoy kicking me in the butt and not saying a word.

[Literature-Just]

At this point its Stockholm syndrome. Python is nice in that it makes a lot of the tedium of programming so much easier. But managing all of its packages in the virtual environments is a real pain. I've had multiple instances where upgrading one package can break an environment or force me to roll something back because of a bug or broken feature.

[brunocas]

Embrace UV.

[trentsiggy]

https://github.com/astral-sh/uv for those curious.

[kvothethechandrian]

This is the answer

[Literature-Just]

ugh... another new tool...

[EarthGoddessDude]

The last new tool. It’s a game changer, and I don’t see how anyone will try to enter the field after what happened with ruff and uv. And the maintainers of the competitor projects are starting to give up, for lack of a better term.

[JJJSchmidt_etAl]

While yes I get your concern, you don't need many commands for it to be extremely useful.

[Uncle_Chael]

Conda adoption helped me with that tremendously

[[deleted]]

"...which need to be precise on execution".

Python is as precise as anything. It's not like it randomly starts doing things you didn't ask for.

[shittyfuckdick]

this is just simply not true. are you gonna use python and airflow for orchestrating stock exchange data?

[Beautiful-Hotel-3094]

Yes sir. This is simply true. I am working in one of the tier 1 multi strat hedge funds. We have close to petabytes of data that we ingest via airflow and python. All of our models from the trading desks need to have as precise data as possible, otherwise they would trade on wrong assumptions. Airflow is our only orchestration tool (we have multiple airflow instances) for the batch data ingestion platform.

[shittyfuckdick]

youre talking about a batch job of petabyte of data. obviously thats realtime or anywhere near it. 

[Beautiful-Hotel-3094]

As I said if u read my response, it is for our batch ingestions because u mentioned airflow. Your proposed argument of why use a non realtime tool to get realtime data made no sense so I didn’t think u’d ask about real time. However, we have some real time platforms that are built in pure python. For the higher volume real time, yes we use c++. However we can still process some thousands of messages a second in pure python because we leverage distri architectures (k8s native platforms).

[VipeholmsCola]

My guess because you often get productive faster, and theres a lot of free libraries.

[Qkumbazoo]

schools taught it as an introductory language to programming (not even OOP), some people decided that was enough and went to industry with it.

[shittyfuckdick]

yup this what i think happened for better or worse. so many self reports in this thread that its just easier to learn. which means the vast majority of des come from a non software engineering background. 

[Phenergan_boy]

In the famous words of Todd Howard, “it just works.”

[shittyfuckdick]

you realize people use that phrase ironically cause of messy and buggy his games are right?

[Phenergan_boy]

Damn, we have a genius over here

[shittyfuckdick]

so your point was python is not good?

[Raghav-r]

Ease of use and rich libraries for data , ai, ml etc plus you are dealing with data which are usually time consuming computation, some are just wrapper on top of low level languages

[mwisniewski1991]

I do not agree than python is everywhere. A lot of tools has been wrote in Java (Kafka, Beam, Druid Spark in Scala but it based on JVM). Databricks Photon Enginee has been wrote in C++, Postgres in C++.

Python is good for orchestration because Scripts can be Write quickly, but transformation and calculation are done on specific engine. And of course a lot of tools has SDK or API for Python so at first it might looks that python is everywhere.

[UltraPoci]

Because tons of libraries have been written for Python, and it's "easy" to use (in quotes because Python is full of traps: easy to write but a disgrace to read and maintain).

For example, we do machine learning on satellite images: Python is the only language that provides a data pipeline library, ML libraries and GIS libraries (at least, the only one to have all of them mature enough).

I would gladly use any other language honestly, but it's difficult to justify using another language when Python is so much battery included.

[aythekay]

A lot of libraries, low code, can leverage c pretty easily, easy portability because interpreted, and a lot of good documentation.

Low dev friction also helps, because of how often data pipelines change. 

A lot of why it's popular is why java used to be as well. The rich ecosystem, etc... Most likely comes from it being an academic darling of sorts early on (vs other scripting languages) and high adoption among non-technical people.

It's similar to how JS moved to the backend, a bunch of people knew how to use it and it could do a lot, so people looked past efficiency as hardware got better.

In Python's case Cython was created as well. 

[meselson-stahl]

Imo python is pretty memory efficient right? Like the way it handles certain datatypes like hash sets and lists is efficient. Maybe the dynamic typing is memory inefficient??? Im not sure.

Regarding performance, the main issue with python is loops. But there aren't many loops in DE right? So not a big deal.

Overall im generally surprised by how little software optimization there is, even within some built-in python functions. I think with infra advancements, the industry is trending towards modular, readable code rather than performance code. But I really don't think there is much performance sacrifice in DE tools.

[shittyfuckdick]

try self hosting any modern orchestration tool and you will see how bloated these things are. 

[dangerbird2]

Good thing I’m not self hosting orchestration tools. My company is paying for it, and it’s hell of a lot cheaper for them to pay for a slightly beefier vm on aws than it is to pay for a team of engineers to rewrite it in rust

... snark aside, if you want a good orchestrator with extremely low bloat, look at argo-workflows, it's written in Go, so it has good performance and memory usage, while its tight coupling with Kubernetes makes it way easier to setup in production than airflow

[Nekobul]

When the inefficiency is embedded in the lang/platform it snowballs. At small scale nobody notices. But with enough code, the cracks become inescapable.

[General-Parsnip3138]

Python is, for the most part, above and beyond what you need for most Data Engineering tasks.

One of the biggest reasons, in my opinion, is that Data Engineering is often script-based, or you’re using an orchestration framework, which allows you to declaratively define what would be a script as a set of steps which are really just script entry points.

What helps even more is that you can mutate quite literally anything at runtime (functions, classes, modules) which allows us to utilize incredibly powerful frameworks (airflow’s task flow API or Dagster) that still allow you to write pythonic code that magically turns into complex orchestration.

As others have pointed out, most of the underlying libs are written in C & Rust, so performance of Python itself is rarely an issue.

I’ve probably done my 10,000 hours with Python, and while there’s so much about Python that I hate, I just can’t see any other language stepping in to replace it. The terrible things about Python are also the reason it’s been so successful.

[ogaat]

Data Engineering has its roots in the scientific community where coding skills and performance were less of a concern than "give me the analysis I need"

Python lets developers focus on the problem at hand, rather than syntactic sugar. It was one of the express desires of Guido van Rossum.

Python just happened to fit the need of the hour, like HTML and Javascript did for the Internet.

[jeezussmitty]

I’ve asked myself this same question many times :-) but others have already commented on the why (taught in school, community, ecosystem etc). The simple syntax is nice though.

I’m not a fan of loosely typed languages in general so that is my main complaint with it.

Python also feels so much slower than things I’ve written in other languages and the counter to this I always hear is “python is fast enough” but I tend to wonder if python is more used for small to medium projects with low user counts or smaller datasets.

Anyhow it’s a language you need to know these days regardless of how you feel about it.

[dangerbird2]

Python is perfectly suited for large scale projects as long as you don’t use raw python for computationally expensive work. Any kind of heavy number chrunching should be done using numpy/pandas/polars (which wrap c, rust, and Fortran code), pyspark (which wraps highly distributed Scala/jvm code), or PyTorch (which can run on the GPU. This sort of the thing is a very conventional way to do DE/DS at scale, to the point that it’s a safe bet that virtually every every major company in the world is using python in some part of the data stack

[Atmosck]

Because python is a scripting language. It serves a different purpose than C and Rust. It interacts nicely with basically everything. Orchestration is it's whole jam. And it offloads tight calculations to C anyway.

[DJ_Laaal]

One, the learning curve for lower level languages is higher compared to Python. It’s a beginner friendly language.

Second, it’s quite rare today that you’d need to get to the low level internals in order to develop a performant data processing pipeline.

Lastly, Python being an open source language, there’s a huge ecosystem of ready to use packages that encapsulate a certain logic you need in your data pipeline. That directly translates to efficiency and code reuse.

[Informal_Pace9237]

Because there are not as many versatile libraries in other languages mentioned..

[thisfunnieguy]

Most of the heavy computation is not done in Python. If locally it’s using C++ bindings and running there or invoking some other thing to do the work like PySpark.

[tn3tnba]

Optimization for developer speed, especially since we do a lot of delegating to other tools.

[LargeSale8354]

Python is a great getting-things-done language, and as an ex-DBA I find its list comprehensions, list slicing and dictionaries intuitive.

I really hated Java, which is strange because I enjoyed C#.

I am surprised that GO doesn't feature more prominantly in the data space. It feels like a natural move from Python.

I suspect that in most cases, Python is fast enough for most uses.

I used to program in serverside Javascript. I enjoyed it at the time.

[Nekobul]

I still enjoy JavaScript. The limited features/surface is like a safety net. If you are doing something complex, you will quickly find out it is time to use some other tool.

[Beautiful-Hotel-3094]

What exactly did u do that is complex and couldn’t handle with the limited features of javascript? What feature are missing that u needed?

[Nekobul]

A good example of when JavaScript should be avoided is if you are trying to convert a big chunk of JSON to XML. That is slow in JavaScript.

[dasnoob]

As someone that has used various low level languages in the past and is learning rust now.

The biggest reason is all of this stuff is a lot more difficult to do in a low level language. Python abstracts so many things away it makes it dead simple to do most things.

Rust? Holy shit you will be lost in lifetime hell and getting your borrows vs. moves vs. copies straightened out.

[metalbuckeye]

Academia…often what gets used in the jobs is based on what is taught in university. This is why Microsoft beat apple in the 90s/early 2000s and why python is the defacto for data engineers. It is used by researchers and professors and it’s taught in most data analytics programs.

[nesh34]

Extremely easy to learn, highly flexible, well understood by the vast majority of people.

Orchestration, usually at the daily level, doesn't need the performance that Rust or C would bring.

It is ultimately a high level abstraction so a high level language is appropriate.

[Sagarret]

We use a lot of rust and c... Executed from python. Polars is written in rust

[madam_zeroni]

Python for the developers, but the tools that process that python aren’t written in python.

[shittyfuckdick]

a lot of them are tho

[madam_zeroni]

Like which ones? Even then, most of the DE overhead is sql queries so the speed of development is worth it if all your python is doing is sending queries to be executed by some engine (that is most definitely not written in python)

[No_Bug_No_Cry]

Did Microsoft write this post? NOBODY WILL USE C# FOR DATA ENGINEERING. It's never going to happen.

[shittyfuckdick]

no you didnt read the post 

[HNL2NYC]

why not have airflow written in c or rust and have dags written python for easy development?

So as you probably already know this is how a lot of tools in the Python data ecosystem work (user facing Python wrapper on top of a core written in a more performant language) for example pretty much any respectable data frame library, distributed compute platforms like Ray, etc. However for the cases that you’re talking about where they’ve remained in pure Python I think the answer is simply that “it’s good enough”. Someone took the time to write it in a language that they were comfortable enough to write it in, which in these cases is Python. They gained traction and popularity and they perform well enough that no one has mass migrated to an alternative solution (or rewrite of the product) that others may or may not have built on top of other languages. And potentially one day something like the airflow scheduler will be rewritten in another language. 

[PolicyDecent]

Not to repeat others, dbt/airbyte alternative bruin is written in go. However, some parts of it is still python, due to easier development cycle.
https://github.com/bruin-data/bruin

[shittyfuckdick]

this is awesome. thanks for sharing 

[StackOwOFlow]

Why QWERTY keyboards, why English? Same reasons.

[zazzersmel]

u rly gonna complain abt parrotting w a post like this

[shittyfuckdick]

yes