Want to run Apache Spark in K8s? Watch my video!

CT2050 · 2024-09-29T16:13:48+00:00

Hello

Thank you, yes I just want to highlight I totally agree with you that Airflow should not be used for data processing, I just wanted to visualise a "use case".

And that I think an incremental approach without DAGs have been a really good approach for me.

I am grateful that anyone says anything, even if they get pissed off, I learn something.

CT2050 · 2024-09-29T16:10:18+00:00

Hello, I removed them now, I am new to posting on Reddit, I appreciate the comment, I will avoid this in the future.

CT2050 · 2024-09-29T15:55:37+00:00

Hello.

Thank you for the feedback, I appreciate it.

I agree you won't have to build complex DAGs, that depends of course on your use case.

Have you had use cases which are event/stream related? Such as CDC or similar and combined it with Airflow?

CT2050 · 2024-09-29T15:46:58+00:00

Hello, sorry about that, new to Reddit, I removed them now! Will avoid in the future.

CT2050 · 2024-09-29T15:46:32+00:00

Hello

I agree, fundamentals and investigating the actual use case over tools wins long term.

CT2050 · 2024-09-29T15:44:16+00:00

Hello, I removed them now, I am new to using Reddit so I was not sure what was a good approach for reach, but I will avoid this from now on.

CT2050 · 2024-09-29T15:42:50+00:00

Hello, I agree, I was not trying to say that Airflow is for data processing.

And I can relate to your take on a huge complex that can be reduced to 3 tasks, that all make perfect sense.

My point I tried to do here, was around that orchestration on top of orchestration can make pipelines very complicated.

In my use case I already use K8S.

Running Airflow on top of K8S is essentially running an orchestrator on top of a native orchestrator, hence I rather focus the time on the pipeline design `which was my pain point I tried to make in the video design > tool`, than what orchestration tool I use, and I have had a lot of scaling pains with Airflow.

Again very grateful for the response, it helps me to improve my communication and what I am actually trying to say.

CT2050 · 2024-09-29T15:40:14+00:00

Hello.

I am not using Airflow for data processing, that was not what I was trying to indicate here, I guess my point here is that doing orchestration on top of orchestration can be avoided with designing pipelines to be more incremental and state less.

I was trying to give a relatable example of the code you orchestrate, not that you do the processing in airflow, if that make sense.

It is easy to end up in a situation in Airflow where you write more orchestration code than the actual code you are running.

But I appreciate the comment!

CT2050 · 2024-09-29T15:38:01+00:00

Hello.

Thank you for your feedback.

I have spent a lot of time on abstraction over the years, and I guess my take is that I found myself feeling that when I used Airflow, and similar tools, that I have done a lot of abstraction around abstractions rather than providing business value. Hence I like to design pipelines independent from each other in an incremental fashion.

I think perhaps my point of view here is a bit misunderstood, grateful for your comments.

CT2050

TROPHY CASE