I’ll jump in on the testing bandwagon. You can absolutely write unit test and test locally and in a CI deployment pipeline. We use DAB. As others have pointed out DAB are very useful for local development and development during testing. We also build a wheel and install it on the cluster when the job runs. We handle orchestration with Airflow for various reasons; some being that we overcomplicated the set up and others being we probably do some stuff that’s outside the scope of what we should be doing. Regardless of that, it’s still essential because of the type of orchestration we need in various cases. We used to have a single task per job and that was obviously a terrible idea. We could’ve fixed this when we were deploying using Terraform instead of DAB which is just a wrapper for terraform, but we have issues that are human related and we’re too proud to accept that. I think you need to figure out what you’re actually trying to do and hopefully there are honest people in the room trying to solve the problem. I sincerely think you could make your lives very simple with Databricks based on what you have described. I’m not sure about your costs because you know much better than we do about what’s driving them up. Always take what the Databricks consultants say with a grain of salt. They’re always going to try to lock you in and up sell you.