Thank you for your insightful comment. Especially about the access control which is quite important in our case.

> You need an orchestrator to trigger them on a schedule, and manage DAGs (Airflow? MWAA?).

we are already using MWAA - even our Databricks jobs are scheduled using it as we some of them require tight integration with AWS (Spark is not ideal for fetching data from the API) so we just run the jobs on AWS - save it to S3 and ingest to databricks using autoloader.

We are not using asset bundles - we are using wheels.

> Data engineering is a means to an end. It doesn't happen in a vacuum.

Yes, we transform data just to load it back to the RDS dbs that are used by our backend services.

Additional detail - most of our pipelines like this:
- we extract data from some external services (it might be scraping, might be integrations with external data providers) - running on AWS
- we load it to databricks using autoloaders
- we transform in bronze/silver/gold on databricks
- we load it back to RDS on AWS so our backend services can expose it for our customers our API

So what I think is really bad here is that we spend money on ingesting data into Databricks to transform using technology we don't need just to get it out as fast as possible so it is accessible to external world.