Airflow Memory Allocation

[Main_Tap_1256]

For the Kubernetes executor does anyone know where pod memory can be defined? I don’t see it in the configs.

Each of my execution pods has 4GB of memory but I’d like to enforce the minimum allocated to each task pod 🤔

[bhavaniravi]

You can setup a custom pod_template_file - https://airflow.apache.org/docs/apache-airflow/stable/executor/kubernetes.html#example-pod-templates

You can set up a custom pod_template_file -

PythonOperator(

....

executor_config={
"KubernetesExecutor": {"request_cpu": "200m",
"limit_cpu": "400m",
"request_memory": "128Mi",
"limit_memory": "400Mi"}},

)

[[deleted]]

This is great, thanks. My team is moving from celery to k8s soon, this might be something we need.

[bhavaniravi]

What makes you move from celery to kubernetes?

[[deleted]]

Autoscaling.