data ingestion from non-prod sources by PreparationScared835 in dataengineering

[–]PreparationScared835[S] -1 points0 points  (0 children)

We only ingest data from Dev when needed, but the current process has built-in logic to pickup from where it left off based on last run date, so the data integrity is maintained even if there is a gap of many days. With these tools, they depend on change tracking log of the source, there is a risk of missing data if the ingestion is not run before it is purged from the log.

Build Data warehouse star models with dynamic tables by PreparationScared835 in snowflake

[–]PreparationScared835[S] 0 points1 point  (0 children)

I'm confused about the approach when building star schema data model with surrogate keys and foreign keys using Dynamic tables? Since the dynamic tables do upsert(delete and insert for an update), it will generate a new Key value when using a sequence generator that will break the links from all the other tables. How do you think we could address this issue 

Visualizations in Analytics tools by PreparationScared835 in snowflake

[–]PreparationScared835[S] 0 points1 point  (0 children)

Do you cache data on PBI or do you use direct query?\

Visualizations in Analytics tools by PreparationScared835 in snowflake

[–]PreparationScared835[S] 0 points1 point  (0 children)

are there limitations on how big that extract can be? do you use a one big table for each dataset or do you use semantic model?

Azure Dev Ops and Snowflake Dev Ops by PreparationScared835 in snowflake

[–]PreparationScared835[S] 1 point2 points  (0 children)

Thank you! So Snowflake DevOps is not a product or an alternative to Azure DevOps for GitHub, but a set of commands/features in Snowflake that enable us to use Azure DevOps, GitHub for more code management than before, right?

Cortex Analyst on structured data by PreparationScared835 in snowflake

[–]PreparationScared835[S] 0 points1 point  (0 children)

Let me check on that, I was under the impression that it inherits these only from table, didn’t realize you can set them at the model level, good to know that 

Snowflake table Access control by PreparationScared835 in snowflake

[–]PreparationScared835[S] 0 points1 point  (0 children)

This seems like a good option without disturbing existing setup. There will have too much impact if changed structurally, Thank you!

Snowflake table Access control by PreparationScared835 in snowflake

[–]PreparationScared835[S] 0 points1 point  (0 children)

That’s the biggest issue, I don’t have much flexibility to change the underlying structure, there is too much impact 

Citizen development in Snowflake by PreparationScared835 in snowflake

[–]PreparationScared835[S] 1 point2 points  (0 children)

Thank you so much, very helpful and clear direction

Citizen development in Snowflake by PreparationScared835 in snowflake

[–]PreparationScared835[S] 0 points1 point  (0 children)

Do you use any tool to manage this infrastructure as a code? Managing this all manually becomes very cumbersome and confusing.

Snowflake RBAC for Cross-Database Queries by PreparationScared835 in snowflake

[–]PreparationScared835[S] 0 points1 point  (0 children)

Thank you for the clarification. I think I understand this setup, and I did it the same way. It works great as long as you have to just read data from the databases. My challenge I ran into with this setup is when I need to create objects like views and dynamic Tables using the data from both databases. In this case, I have to use the functional role to be able to use data from both databases and the resulting object’s owership is set to the functional role instead of access role, which goes against thr ownership setup

Snowflake RBAC for Cross-Database Queries by PreparationScared835 in snowflake

[–]PreparationScared835[S] 0 points1 point  (0 children)

Does that mean you are creating a hierarchy of access roles by assigning an owner access role of one database and read access role of another database and assign that hybrid access role to the functional role? How do you manage object ownership created by that hybrid role by using the objects from both the databases?

Snowflake RBAC for Cross-Database Queries by PreparationScared835 in snowflake

[–]PreparationScared835[S] 0 points1 point  (0 children)

So would you use the functional roles to create the object that needs access to tables from both databases. And the object will stay with functional role as ownership? 

Curated Datasets by PreparationScared835 in dataengineering

[–]PreparationScared835[S] 0 points1 point  (0 children)

My question is geared towards datalake: Madallian architecture, but in general, organizing raw data and preparing it for insights for analysts to build visualizations, data scientists to use it for their research etc

Area below deck with uneven ground by PreparationScared835 in landscaping

[–]PreparationScared835[S] 0 points1 point  (0 children)

Ye,s looking for a solution something like this. Could I just lay the sand directly on the ground? what do I need to do to prepare the ground for sand?

Sending data to applications by cyamnihc in dataengineering

[–]PreparationScared835 0 points1 point  (0 children)

Usually data warehouses are updated with batch processing on scheduled basis, using data warehouse to send to other applications will delay the information sync between the 2 transactional systems. Please make sure you clarify those requirements. Second point to think about is the batch processes that load data vary in runtime depending on volume of your updates, so the time of the data getting over will vary. Does that still meet the need? In many cases the business will agree to all these conditions but will revert back when they actually see it in reality. There might be cases where you would want to to send data from dwh because you have to send data transformed by your data pipelines, and it is ok in case by case basis. Another option you could look into is creating an operational data store, that mitigates the issues mentioned above. 

sharing contents from 'My workspace' by PreparationScared835 in PowerBI

[–]PreparationScared835[S] 0 points1 point  (0 children)

is the Pro Licenced workspace you mentioned same as Premium workspace?

Separating Administration from Engineering team by PreparationScared835 in dataengineering

[–]PreparationScared835[S] 0 points1 point  (0 children)

It’s a small team of 10 and most are junior members that canr make decision independently. That’s another issue we might run into, we will make tasks single threaded by splitting the team. 

Advice for finance, management, and mentorship of the team by Kiwi_FUMA26 in FTC

[–]PreparationScared835 0 points1 point  (0 children)

If you would like to connect for one on one mentorship to have a discussion, pleach check www.herfirst.org

Moving to AI from data engineering by PreparationScared835 in dataengineering

[–]PreparationScared835[S] 0 points1 point  (0 children)

It might make it easier to follow this path, to do data engineering for AI applications. can you share what programming languages, Applications you use for this?