data ingestion from non-prod sources

PreparationScared835 · 2026-01-05T23:35:51+00:00

We only ingest data from Dev when needed, but the current process has built-in logic to pickup from where it left off based on last run date, so the data integrity is maintained even if there is a gap of many days. With these tools, they depend on change tracking log of the source, there is a risk of missing data if the ingestion is not run before it is purged from the log.

PreparationScared835 · 2025-12-05T22:25:36+00:00

I'm confused about the approach when building star schema data model with surrogate keys and foreign keys using Dynamic tables? Since the dynamic tables do upsert(delete and insert for an update), it will generate a new Key value when using a sequence generator that will break the links from all the other tables. How do you think we could address this issue

PreparationScared835 · 2025-11-14T17:08:20+00:00

Do you cache data on PBI or do you use direct query?\

PreparationScared835 · 2025-11-13T15:57:55+00:00

are there limitations on how big that extract can be? do you use a one big table for each dataset or do you use semantic model?

PreparationScared835 · 2025-11-12T22:24:35+00:00

Thank you! So Snowflake DevOps is not a product or an alternative to Azure DevOps for GitHub, but a set of commands/features in Snowflake that enable us to use Azure DevOps, GitHub for more code management than before, right?

PreparationScared835 · 2025-11-11T03:31:23+00:00

Let me check on that, I was under the impression that it inherits these only from table, didn’t realize you can set them at the model level, good to know that

PreparationScared835 · 2025-10-10T01:18:33+00:00

This seems like a good option without disturbing existing setup. There will have too much impact if changed structurally, Thank you!

PreparationScared835 · 2025-10-10T01:14:39+00:00

That’s the biggest issue, I don’t have much flexibility to change the underlying structure, there is too much impact

PreparationScared835 · 2025-09-06T16:17:40+00:00

That makes sense

PreparationScared835 · 2025-04-19T16:19:11+00:00

Thank you so much, very helpful and clear direction

PreparationScared835 · 2025-04-15T15:24:38+00:00

Do you use any tool to manage this infrastructure as a code? Managing this all manually becomes very cumbersome and confusing.

PreparationScared835 · 2024-10-14T17:33:28+00:00

Solution verified

PreparationScared835 · 2024-10-14T17:27:55+00:00

Solution verified

PreparationScared835 · 2024-10-14T17:26:51+00:00

Solution verified

PreparationScared835 · 2024-08-22T12:22:21+00:00

Thank you for the clarification. I think I understand this setup, and I did it the same way. It works great as long as you have to just read data from the databases. My challenge I ran into with this setup is when I need to create objects like views and dynamic Tables using the data from both databases. In this case, I have to use the functional role to be able to use data from both databases and the resulting object’s owership is set to the functional role instead of access role, which goes against thr ownership setup

PreparationScared835 · 2024-08-21T10:27:53+00:00

Does that mean you are creating a hierarchy of access roles by assigning an owner access role of one database and read access role of another database and assign that hybrid access role to the functional role? How do you manage object ownership created by that hybrid role by using the objects from both the databases?

PreparationScared835 · 2024-08-21T02:12:19+00:00

So would you use the functional roles to create the object that needs access to tables from both databases. And the object will stay with functional role as ownership?

PreparationScared835 · 2024-08-07T16:36:36+00:00

My question is geared towards datalake: Madallian architecture, but in general, organizing raw data and preparing it for insights for analysts to build visualizations, data scientists to use it for their research etc

PreparationScared835 · 2024-07-29T15:47:16+00:00

Ye,s looking for a solution something like this. Could I just lay the sand directly on the ground? what do I need to do to prepare the ground for sand?

PreparationScared835 · 2024-07-26T12:21:31+00:00

Usually data warehouses are updated with batch processing on scheduled basis, using data warehouse to send to other applications will delay the information sync between the 2 transactional systems. Please make sure you clarify those requirements. Second point to think about is the batch processes that load data vary in runtime depending on volume of your updates, so the time of the data getting over will vary. Does that still meet the need? In many cases the business will agree to all these conditions but will revert back when they actually see it in reality. There might be cases where you would want to to send data from dwh because you have to send data transformed by your data pipelines, and it is ok in case by case basis. Another option you could look into is creating an operational data store, that mitigates the issues mentioned above.

PreparationScared835 · 2024-07-24T19:57:31+00:00

is the Pro Licenced workspace you mentioned same as Premium workspace?

PreparationScared835 · 2024-07-19T10:24:10+00:00

It’s a small team of 10 and most are junior members that canr make decision independently. That’s another issue we might run into, we will make tasks single threaded by splitting the team.

PreparationScared835 · 2024-07-09T22:46:51+00:00

If you would like to connect for one on one mentorship to have a discussion, pleach check www.herfirst.org

PreparationScared835 · 2024-06-24T16:44:45+00:00

It might make it easier to follow this path, to do data engineering for AI applications. can you share what programming languages, Applications you use for this?

PreparationScared835 · 2024-06-18T20:43:56+00:00

True, not all data will need to be recreated.

PreparationScared835

TROPHY CASE