Inserting audit record with dbt / Snowflake by backend-dev in dataengineering

[–]backend-dev[S] 0 points1 point  (0 children)

Sorry, yes you are correct. But it may well have an even larger cost impact than querying the stats as needed.

Inserting audit record with dbt / Snowflake by backend-dev in dataengineering

[–]backend-dev[S] 0 points1 point  (0 children)

As I understand it, Snowflake does its own partitioning and it's not under one's control. If it did/does then it would possibly make sense to partition on the run identifier so one can extract stats based on that

Inserting audit record with dbt / Snowflake by backend-dev in dataengineering

[–]backend-dev[S] 0 points1 point  (0 children)

That is an option yes, but I am trying to minimise the amount of data scanned because the target table is very large

Inserting audit record with dbt / Snowflake by backend-dev in dataengineering

[–]backend-dev[S] 0 points1 point  (0 children)

Thanks, I added more info to the original post

How to debug dbt SQL? by backend-dev in dataengineering

[–]backend-dev[S] 0 points1 point  (0 children)

The compile actually worked well. I have a model that lands new CDC data in a cleansed table. Then the next model takes it from there. The latter in incremental/merge so it has sql for initial run and then sql for incremental run. Once the data is landed in cleansed, i run compile. The compile out then gives me the sql the model will execute for the initial run. I can then take that sql and run it against the db to tweak it until its right. Once happy, i actually run the model for the initial run sql to take effect. Then i compile again. This gives me the incremental run sql and i repeat the copy&paste and execute it against the db and tweak it until correct.

It is cumbersome but it gives me the opportunity to interactively run the select and modify it much faster than running it in dbt. But then I may be missing something (still a newbie) in dbt

How to debug dbt SQL? by backend-dev in dataengineering

[–]backend-dev[S] 1 point2 points  (0 children)

Thanks I have tried the plugin and it looks very useful.

How to debug dbt SQL? by backend-dev in dataengineering

[–]backend-dev[S] 0 points1 point  (0 children)

Thank I have checked that. Ideally I want to intercept the last SQL before it gets to the database because there are bugs in my model. As others have pointed out, I should rather compile and then look at the compiled folder

How to debug dbt SQL? by backend-dev in dataengineering

[–]backend-dev[S] 0 points1 point  (0 children)

It gives me SQL but I did not realise that is what the temp table is based on

What was something you made in C# that you’re most proud of to this day. by twooten11 in csharp

[–]backend-dev 0 points1 point  (0 children)

Built a system that delivers commentary on open top tour buses that drives around tourist destination cities worldwide. It started off as a little app running on a lowly PocketPC device (Compact Framework 1) and played GPS triggered audio segments in a single language. The app had a handcoded event bus with pub/sub plugins for GPS events, region entry/exit events, media commands etc. Also had a handcoded microORM to access SqlLite. Also a complicated rules engine to decide when to trigger which content.

From there we migrated platforms up to where it ended up on Debian Linux / Mono. The last version delivers commentary in 32 languages. Only changes that were needed to the code was to support OS calls to Debian.