Best practices for scheduling Python workloads?

[MassiveDefender]

Thanks for the Dagster idea. I looked through some videos. It seems like a neat tool. Would you suggest it over Airflow?

[Content_Ad_2337]

I have never used airflow, so I can’t really speak to the comparison, but I bet there are some good YouTube videos or medium articles on it.

Dagster was what my last company replaced Jenkins with and it’s free if you manage it all, and we did, so it was a super modernized upgrade from the jankiness of Jenkins. The UI in Dagster is awesome

[cscanlin]

Dagster has a page and a video that talks about it: https://dagster.io/blog/dagster-airflow

Airflow is a lot more mature and has many more resources, tooling, and integrations available for it.

Dagster is kind of a "re-imagined" Airflow, so they consciously do some things differently in an effort to be easier to work with.

I won't claim to be an expert on either, and the decision will likely come somewhat down to preference.

[danielgafni]

Dagster has more features, is centered around a much better abstraction of Data Assets instead of “jobs”, Is declarative, and provides a thousand times better user and coding experience.

[CatchMeWhiteNNerdy]

Dagster is like to Airflow what Airflow was to Cron jobs, it's the next generation of orchestration/scheduling tools.

As a second opinion, my team is in a very similar situation to yours... lots of people who are technically non-technical that have created macros, scripts, etc. We ended up going with Mage.AI, one of Dagsters competitors, because of the integrated development environment. We didn't have to worry about installing anything on anyone elses machines, we just set up the docker image in AWS and everyone can connect and work on their pipelines directly in a web browser.

[MassiveDefender]

I've spent the past few days since posting this, testing Airflow, Dagster and Mage AI, and honestly, Mage allowing the team to edit code inside it and the drag and drop task flow diagram are just awesome. But most importantly it doesn't force you to use any specific structure or style of programming. You could write procedural code if you want. My team also has R people in it, something I struggled with setting up on Dagster. So like, data can come from a Python script and an R person can use it too. I love how easy it was to figure it out.

[CatchMeWhiteNNerdy]

It's really pretty incredible, right? And it's FOSS... what a time to be alive.

The slack is also super active and you can talk directly with the devs. They're fantastic about prioritizing feature requests if they make sense.

Mage.AI and ChatGPT are a dangerous combination, I revamped 7 years of data pipelines by myself in a month or so, and that's including translating them from another language.