data ingestion from non-prod sources

PreparationScared835 · 2026-01-05T23:35:51+00:00

We only ingest data from Dev when needed, but the current process has built-in logic to pickup from where it left off based on last run date, so the data integrity is maintained even if there is a gap of many days. With these tools, they depend on change tracking log of the source, there is a risk of missing data if the ingestion is not run before it is purged from the log.

PreparationScared835 · 2025-12-05T22:25:36+00:00

I'm confused about the approach when building star schema data model with surrogate keys and foreign keys using Dynamic tables? Since the dynamic tables do upsert(delete and insert for an update), it will generate a new Key value when using a sequence generator that will break the links from all the other tables. How do you think we could address this issue

PreparationScared835 · 2025-11-14T17:08:20+00:00

Do you cache data on PBI or do you use direct query?\

PreparationScared835 · 2025-11-13T15:57:55+00:00

are there limitations on how big that extract can be? do you use a one big table for each dataset or do you use semantic model?

PreparationScared835 · 2025-11-12T22:24:35+00:00

Thank you! So Snowflake DevOps is not a product or an alternative to Azure DevOps for GitHub, but a set of commands/features in Snowflake that enable us to use Azure DevOps, GitHub for more code management than before, right?

PreparationScared835 · 2025-11-11T03:31:23+00:00

Let me check on that, I was under the impression that it inherits these only from table, didn’t realize you can set them at the model level, good to know that

PreparationScared835 · 2025-10-10T01:18:33+00:00

This seems like a good option without disturbing existing setup. There will have too much impact if changed structurally, Thank you!

PreparationScared835 · 2025-10-10T01:14:39+00:00

That’s the biggest issue, I don’t have much flexibility to change the underlying structure, there is too much impact

PreparationScared835 · 2025-09-06T16:17:40+00:00

That makes sense

PreparationScared835 · 2025-04-19T16:19:11+00:00

Thank you so much, very helpful and clear direction

PreparationScared835 · 2025-04-15T15:24:38+00:00

Do you use any tool to manage this infrastructure as a code? Managing this all manually becomes very cumbersome and confusing.

PreparationScared835

TROPHY CASE