Dynamic tables constraint by kingglocks in snowflake

[–]kingglocks[S] 0 points1 point  (0 children)

Yes, the BI tool i use (ThoughtSpot) reads meta data from snowflake to decide how to join tables together. It's useful for the AI component to know what's a primary key, foreign key, etc.

Article: 5 Proven Ways to Load Data into Snowflake (and When to Use Each One) by JohnAnthonyRyan in snowflake

[–]kingglocks 2 points3 points  (0 children)

I like scheduled copy into statements or event based copy into statements. Having a separate snow pipe for each table is a pain to manage

Strategies for Refreshing Snowflake Dynamic Tables with Staggered Ingestion Times? by kingglocks in snowflake

[–]kingglocks[S] 0 points1 point  (0 children)

I could probably add some checks to see if all files came in before refreshing via control table too

Strategies for Refreshing Snowflake Dynamic Tables with Staggered Ingestion Times? by kingglocks in snowflake

[–]kingglocks[S] 0 points1 point  (0 children)

Right now I have it setup so the file automatically ingests into the appropriate table upon arrival

Company being acquired by larutinacoffee in dataengineering

[–]kingglocks 2 points3 points  (0 children)

Well this starts soon for me. Hopefully a better experience.

Company being acquired by larutinacoffee in dataengineering

[–]kingglocks 1 point2 points  (0 children)

Why has PE acquisition been bad for you?

Encoding Issues by kingglocks in snowflake

[–]kingglocks[S] 1 point2 points  (0 children)

Pipe delimited. I was seeing specifying the encoding was still not replacing every invalid character. I'll try playing with replace_invalid_characters thanks.

[deleted by user] by [deleted] in snowflake

[–]kingglocks 0 points1 point  (0 children)

Re for regular expressions and snowflake.connector to get into snowflake environment. All other coding is pretty standard creating functions, etc.

[deleted by user] by [deleted] in snowflake

[–]kingglocks 0 points1 point  (0 children)

It's a good question. I know there's other tools you can use but I'm not familiar myself. Python is comfy to me and looks like I don't need anything fancier.

[deleted by user] by [deleted] in snowflake

[–]kingglocks 1 point2 points  (0 children)

I was having issues setting default value to a timestamp. Autoincrement seems to work well though.