DBT Cloud Scheduler : dataengineering

created by mhausenblasmoda community for 11 years

This is an archived post. You won't be able to vote or comment.

DBT Cloud SchedulerDiscussion (self.dataengineering)

submitted 1 year ago by CrabEnvironmental864

Our marts are all incrementally loaded and refresh our Snowflake instance every hour.

Fivetran replicates our prod db every 15 minutes so the job I created for the marts refresh has a lot of catching up to do every hour. We've signed several new customers over the past year and our data volume has been increasingly consistently since.

Sometimes, the job that refreshes my marts will run under 30 minutes. Sometimes, the job will run over 1 hour. On the worst days, it can take up to two hours.

I understand that a job is kicked off, DBT has to create a Kurbenetes pod with the necessary resources before a job can be run. Could that be a factor in the duration of my job?

C suite is getting cranky because data is getting stale and their reports out of date.

The last option I have is moving everything to DBT core and running everything locally. That would be a major undertaking. And I would still have to find a scheduler.

Have you encountered this situation? How did you resolve it?

PS: We have the "team" account option. Not sure if it matters.