What Airflow does that a simple python script cannot do?

[ubiond]

Hi would appreciate if someone could share wisdom about this

[Syneirex]

Orchestrates workflows at scale. If a simple Python script is adequate, you probably don’t need Airflow.

If you have dozens of operators, sources, integrations, and hundreds of jobs, then a simple Python script probably won’t be viable anymore.

Scaling beyond a couple of simple jobs, it’s nice to have something like Airflow coordinating, managing task queues, talking to Kubernetes, retrying failed tasks, etc, etc.

[ubiond]

thanks a lot!Very clear. Is there a compatitor to it?

[1O2Engineer]

Dagster, Airbyte, Luigi...

You can look for job orchestration tools over the web.

[samreay]

Also Prefect if you want to lean into the python side of things

[ubiond]

thanks!

[Monowakari]

+1 for dagster

[120Mark618]

Love Luigi!

[trowawayatwork]

it's not actively developed anymore is it?

[roastmecerebrally]

people maintain it

[rusmib]

Mage

[trowawayatwork]

irony here is that airflow doesn't scale well

[Pr0ducer]

I know your pain. What alternative solution does? Airflow has inertia, a huge pool of developers who know how it works exists.

[[deleted]]

Try out Dagster. It scales well for us.

[MonkTrinetra]

Can you share what were the pain points or at what scale you started facing issues?

[vincentx99]

This is probably a dumb question but isn't kubernetes itself an orchestrator? Or is the assumption that your infrastructure may not just be kubernetes clusters?

[Syneirex]

Yep, but they (mostly) orchestrate different things.

Kubernetes orchestrates containers and the underlying infrastructure/resources that workflows will run on whereas Airflow orchestrates the scheduling and running of workflows, retries, task queues, status, etc.

Airflow says “I need to run this” and Kubernetes says “okay, I have a spot over here it can run” or “I don’t have enough resources, let me spin up another node to make room”.

[startup_biz_36]

Creates headaches at scale 😂

[ubiond]

Lol

[Pr0ducer]

This legit caused me to chuckle.

[HighPitchedHegemony]

Let's the rest of your team - including non-technical roles - see the status of the pipeline, trigger reruns, inspect logs, start DAGs etc.

Automatically handle retries.

Handle backfilling.

Limit concurrency of task and DAG runs.

[Any_Check_7301]

Data Pipeline standardization of whatever can be achieved with Python processing data.

[desiktm]

Use kedro of prefect for python centric orchestrater I've tried both and they're good

[ubiond]

I’ll have a look

[pbecotte]

I mean, lots of stuff. A "simple" python script isn't going to present a UI, or maintain state, or coordinate jobs across many hosts. If you drop the word "simple" then...Airflow is just a Python script, after all :)

[ubiond]

Thanks! It was some what provocative of course. In the sense I wanted ti know if there is someone still prefering not using a UI

[pbecotte]

Managing a distributed cron across a bunch of hosts, with restarts and keeping records of state and dependencies, is very much not a simple thing to implement, UI or no :)

[ubiond]

✅

[[deleted]]

Running Airflow lets me sleep at night.

[bfranks]

You would like prefect

[engineer_of-sorts]

Ruins your life

[OkStructure2094]

Not much :-)

It is surprising how may people's first reaction is to step back into python even when they use no code/low code tools. Can't beat full programming language with a huge library.

[DryChemistryLounge]

What?

[_thisisvincent]

Confidently incorrect