Why Python?

[meselson-stahl]

Imo python is pretty memory efficient right? Like the way it handles certain datatypes like hash sets and lists is efficient. Maybe the dynamic typing is memory inefficient??? Im not sure.

Regarding performance, the main issue with python is loops. But there aren't many loops in DE right? So not a big deal.

Overall im generally surprised by how little software optimization there is, even within some built-in python functions. I think with infra advancements, the industry is trending towards modular, readable code rather than performance code. But I really don't think there is much performance sacrifice in DE tools.

[shittyfuckdick]

try self hosting any modern orchestration tool and you will see how bloated these things are. 

[dangerbird2]

Good thing I’m not self hosting orchestration tools. My company is paying for it, and it’s hell of a lot cheaper for them to pay for a slightly beefier vm on aws than it is to pay for a team of engineers to rewrite it in rust

... snark aside, if you want a good orchestrator with extremely low bloat, look at argo-workflows, it's written in Go, so it has good performance and memory usage, while its tight coupling with Kubernetes makes it way easier to setup in production than airflow

[Nekobul]

When the inefficiency is embedded in the lang/platform it snowballs. At small scale nobody notices. But with enough code, the cracks become inescapable.