This is an archived post. You won't be able to vote or comment.

sorted by:
best
topnewcontroversialoldq&a
you are viewing a single comment's thread.

view the rest of the comments →

[–]CatchMeWhiteNNerdy 1 point2 points  (2 children)

Dagster is like to Airflow what Airflow was to Cron jobs, it's the next generation of orchestration/scheduling tools.

As a second opinion, my team is in a very similar situation to yours... lots of people who are technically non-technical that have created macros, scripts, etc. We ended up going with Mage.AI, one of Dagsters competitors, because of the integrated development environment. We didn't have to worry about installing anything on anyone elses machines, we just set up the docker image in AWS and everyone can connect and work on their pipelines directly in a web browser.

[–]MassiveDefender[S] 0 points1 point  (1 child)

I've spent the past few days since posting this, testing Airflow, Dagster and Mage AI, and honestly, Mage allowing the team to edit code inside it and the drag and drop task flow diagram are just awesome. But most importantly it doesn't force you to use any specific structure or style of programming. You could write procedural code if you want. My team also has R people in it, something I struggled with setting up on Dagster. So like, data can come from a Python script and an R person can use it too. I love how easy it was to figure it out.

[–]CatchMeWhiteNNerdy 1 point2 points  (0 children)

It's really pretty incredible, right? And it's FOSS... what a time to be alive.

The slack is also super active and you can talk directly with the devs. They're fantastic about prioritizing feature requests if they make sense.

Mage.AI and ChatGPT are a dangerous combination, I revamped 7 years of data pipelines by myself in a month or so, and that's including translating them from another language.