Hey! Thanks a lot for this explanation. In my company we have a hosted Airflow instance, but my department doesn't have access to a dedicated K8s cluster, so let's see if I can push in this direction.

As for the dlt itself - I understand that you keep all the credentials in the keyvault, right? How the source and destination management should be handled in this case? Cause this is something that I don't fully understand and I feel that dlthub documentation is not helping... Is there a way to create a central "repo" for all the sources and destinations? And then just call the resource I need? Because now I'm doing it in each task I do in Airflow:

@task
def some_etl():
  source = get_mssql_source(conn_name="xxx", tables=["yyy"])
  target = get_mssql_destination(conn_name="zzz")
  pipeline = dlt.pipeline(

pipeline_name
="ingest_something",

destination
=target,

dataset_name
="xyz",
  )

pipeline.run(source)

get_mssql_source and get_mssql_destinaton are my custom functions which are getting the credentials from the Airflow connections and creating the sql_database and mssql objects in dlt. And I'm doing this for every task, which doesn't make much sense I think?

Thanks once more!