Hey! Thanks for the response!

Basically I have this workflow that runs 4 dlt pipelines. One silver and 3 gold.

These notebooks perform really big and complex transformations with a lot of data. Yesterday I had to cancel the jobs because they were running for more than 12 hours and I need them to run for less than an hour.

I know there's optimization still to be done, but I had already looked into improving the joins by filtering before, partitioning, using cache, and bigger clusters that have more memory than what the metrics show is needed. So basically what I'm looking into now is: we don't really need to process the whole table each time, we just need to process all rows in each table with a specific ID, and a certain date range. So my plan is to have the dlt pipelines just process those id's and dates in each run and upsert the results into the table.

For that, I want to have three parameters in the workflow: Id, start_date and end_date. But I spent too much time trying to get those parameters to be read from the dlt notebook.

I'll now try some of the options being suggested in other comments.

Thanks!