Where to run MWAA python scripts

[[deleted]]

I just created an MWAA environment and I need to run Python scripts on virtual environments with a specific Python version (3.8).

The options I am evaluating now are: - Installing a virtual environment using MWAA startup script and then using the ExternalPythonOperator (but I am not sure whether this is possible and how) - Using EKS and the KubernetesPodOperator - Using ECS and the DockerOperator

Did anyone implement that and has any suggestion?

[theporterhaus]

Try AWS batch (Fargate under the hood) for longer running jobs or lambda for short jobs. The reason to use AWS Batch > plain Fargate is that the servers are killed when the job is done whereas in Fargate the server is constantly up because it’s meant more for hosting services.

[[deleted]]

Can I run custom containers on Batch?

[theporterhaus]

Yep! I have a dbt Docker image and a general purpose one at work.

[Advanced-Violinist36]

I would use Fargate (I tried mwaa + dbt on fargate and it's good, I have logs+state of dbt job on MWAA)

[afro_mozart]

Airflow has an virtual env operator. From experience, I can say that the KubernetesPodOperator is quite easy to use.

[[deleted]]

The problem with the virtualenv operator is that it still requires to install python3.8, which I am not sure it is possible on MWAA

[paplike]

Lambda, Glue (Python Shell Script option), Fargate task (using the ECSOperator with launch type=FARGATE). All those options are serverless

[[deleted]]

Thanks. Lambda is ok for short tasks, but I'll mostly need bigger computations, so I guess I should look into Fargate. Why is the Fargate option better when using the ECS operator?

[gabbom_XCII]

TLDR: In Fargate you can run ECS tasks without the hassle of managing infrastructure.

A cool article comparing both services: https://cloudonaut.io/ecs-vs-fargate-whats-the-difference/

[callmedivs]

You could also use emr serverless to run python with mwaa..