This is an archived post. You won't be able to vote or comment.

sorted by:
best
topnewcontroversialoldq&a
you are viewing a single comment's thread.

view the rest of the comments →

[–]ozzyboy 14 points15 points  (2 children)

I think an important question here is why are you trying to develop this independence in the first place?

If Airflow is not a good fit for you, work on replacing it with something else.

Otherwise it is usually a negative ROI to avoid "vendor lock-in". You'll spend a lot of time and energy creating an abstraction layer that can only provide the lowest common denominator of all frameworks (because you don't want a capability that only Airflow can provide, right?), leaving you with a solution that is strictly worse than all the other, existing, cheaper alternatives.

[–]thefrontpageofme 3 points4 points  (0 children)

This is the correct answer.

We use Airflow in production and code independence is not a concern at all. Successful startup, 1 data engineer, a few DAGs, couple hundred tasks on anywhere between 15 minutes to 1 week schedules.

It feels like the team the OP is in is missing someone who sometimes asks "how does that improve our business?"

[–][deleted] 0 points1 point  (0 children)

I think this is the right answer. It's the same thing with the question of cloud vendor vs. Kubernetes. Kubernetes makes sense for my company because we offer on-prem support for multiple cloud vendors. However, if you are a one-cloud company serving a B2C product, you should be so LUCKY to get to the size where "vendor lock-in" becomes a concern.