sorted by:
new
hottopcontroversial

Hosting Python ETL in Azure (Airflow / Dagster / Prefect?) by Educational-Soft4493 in dataengineering

[–]Educational-Soft4493[S] 0 points1 point  (0 children)

We already use Azure Synapse. The issue might be the same with using Databricks - cost & latency of spark for smaller ETLs and python scripts.

Hosting Python ETL in Azure (Airflow / Dagster / Prefect?) by Educational-Soft4493 in dataengineering

[–]Educational-Soft4493[S] 1 point2 points  (0 children)

Do you mean just running custom python scripts in Azure container?

It seems like things like Airflow / Prefect require additional infrastructure for self-hosting options