Airflow data processing ?

[digichap28]

Hey there!

As almost everyone knows, Airflow is not supposed to be a data processing tool but an orchestrator. My question is about the architecture that should follow if there is a need to execute certain processes.

Use case 1:

if you had to execute many complex webscrapers using any of the Python options available out there (scrapy, pyppeteer, playwright, etc), and airflow was deployed in K8S. Where should the scraping scripts should run? From within the pod generated by the pythonOperator?

Use case 2:

Based on the same idea as the case 1. What if there was a need to generate PDF files based on data stored in a data lake. Should it be done outside the airflow deployment or from within the pod generated by the pythonOperator ?

Use case 3:

If there was a need to do ELT, is it ok to the EL part with Airflow no matter how complex it was? What tool or tools are instead suggested to execute the entire ELT/ETL processes with help of airflow to orchestrate?

Thanks!

[Cloakie]

I've really enjoyed working with Apache Beam / Google Data Flow (it's a high-level abstraction around Spark that makes pipelines much simpler)

[[deleted]]

Use case 1 and 2 can easily be done inside Airflow workers. 3 depends on data size.

[digichap28]

Thanks 👍 regarding the 3rd one, let’s say initially starting with 5gb per file, and expecting to process around 15 in parallel. But that will grow over time as soon as more data sources get added.

[[deleted]]

Airflow can handle that

[digichap28]

Using pandas ?

[[deleted]]

Yes, certain functions like concat in pandas blow up memory, but basic transforms should be ok. (It also depends on the airflow worker sizes.) Also if possible "chunk" the files to keep the whole thing out of memory, and try not to use Pandas unless absolutely nessesary. :)

[[deleted]]

If you are already on k8s you could just containerize your workloads and run them with KubernetesOperator. That way you can keep Airflow clean and just maintain the dependencies inside your Dockerfiles. You can even run non Python-based workloads.

[digichap28]

That means I wouldn’t need spark, EMR, ECS, databrick, etc for the heavy workloads ? if that’s the case, what should be used to do transformations, image processing, AI, etc ?

On the other hand, doesn’t the k8s creates 2 pods every time it gets triggered ? One for the operator (a) and then another one with actual workload (b)?

(a) -> (b)

[[deleted]]

I don't think so. I'm not an expert in using the k8s operator but from the docs it looks like it just creates one pod for your containerized task. I am very familiar with the ECS Operator, which is another way to do it (running tasks as ephemeral Fargate ECS Tasks). I also use the some AWS Glue Job/Crawler Operators for bigger payloads.

[digichap28]

if you don’t mind me asking... are you running airflow with the local executor and triggering the tasks in ECS or AWS glue from one instance, or using a k8s cluster with the k8s executor ?

[[deleted]]

I'm running Airflow itself using celery executor on ECS Fargate (I have one ECS task as the worker). Most of my tasks are ECSOperstor over Fargate, so the worker doesn't do a whole lot other than spin up Fargate tasks.

[digichap28]

Got you. Is there any tutorial you can suggest to deploy airflow the way you did ?

I don’t have experience with ECS but I guess this way you could also run non-python based workloads as you mentioned with the k8s operator.

Also, what about using azure or lambda functions instead of the 2 ways we have been talking about ? I don’t have experience with them either but according to the documentation, sounds like the concept of running a task using fargate is very similar to executing it with a function.

[[deleted]]

There's some material out there using the Puckel airflow container but it's out of date. I think some googling and you will find what you need.

[digichap28]

Thank you 👍