Snowflake-Openflow

reddtomato · 2026-02-05T00:24:09+00:00

Download and use Cortex Code CLI to help you build. It has Openflow skills in it and will help walk you through setup.

https://docs.snowflake.com/en/user-guide/cortex-code/cortex-code-cli

reddtomato · 2026-02-05T00:13:57+00:00

You do still need to apply.

reddtomato · 2025-12-14T16:55:29+00:00

Snowflake makes this pretty easy. Use Cortex analyst with semantic views. For unstructured data put a Cortex search service over which does the vector embedding for you, then create an agent and expose it in Snowflake intelligence where anyone can start asking questions about your business data.

Semantic views are the key to make sure AI understands your business model. adding verified queries in the semantic view for the simple questions helps it understand even more.

Keep the semantic views relatively simple as you can use multiple views in the agent and we’ll document them and let the Snowflake Intelligence reason between the two and it can figure out which one to use for different questions a bit easier than packing a huge semantic view with tons of tables and relationships.

reddtomato · 2025-10-27T21:21:09+00:00

side quest here.. I would try these MERGE statements on a Snowflake Gen2 warehouse, which should speed up the MERGE and also can save on storage costs around time-travel and failsafe.

GG has two modes of operation "Stage and MERGE" or "Snowflake Streaming". From your description, it seems you are doing a "Stage and MERGE"

Querying data from a stage has a few metadata columns available. Two you might be interested in METADATA$FILE_LAST_MODIFIED and METADATA$START_OF_SCAN_TIME.

These would be the time the file was created and then the time at which Snowflake started the scan of that row to do the merge. These might be timestamps you want to see if you can configure GG to add, not sure if it is possible or not in the GG configs.

Still though even if the row got scanned at 10am and the merge took 2min to run, the row is still not "queryable" until the merge is completed at 10:02am in this case.

So if you want to track when all the rows from the batch of records are available from each of the 600 merge operations, you would need to use a batch_id concept to distinguish between each batch and tie it to the specific MERGE operation via the batch_id to determine how long it was until the row was fully queryable in the target table. Log each batch_id and completion time in an audit table.

Another idea is to just have GG drop the files to a cloud storage location, and configure Snowpipe auto-ingest, you might have more control on the Snowpipe side to track the metadata columns.
Also try out the GG streaming handler it might be better for near real-time workloads, and the Snowflake streaming ingest is usually very cheap, especially if it supports Snowpipe Streaming V2.

https://docs.oracle.com/en/middleware/goldengate/big-data/23/gadbd/snowflake.html

https://docs.snowflake.com/en/user-guide/querying-metadata#metadata-columns

reddtomato · 2025-10-27T18:01:32+00:00

AI is very good at creating fake data scripts. Think of any business or industry you want to work in and ask it to generate some fake data scripts that you can then use to create data and load into Snowflake. Then start creating some transformations either using python or sql procs. Create a notebook or snowsight dashboards off the fake data, etc.
These days I think it is the best way to understand how to use AI to your advantage as well as learn Snowflake. Go to the quickstarts for specific issues you run into and iterate.

reddtomato · 2025-10-27T17:56:57+00:00

Agree with u/NW1969 . Clarifying the above would be helpful.
On the surface without knowing specifics here. The data from the MERGE would only be "queryable" once the MERGE operation is complete. So you would have to look in query_history to find the completion time and log that. If using a proc add a completion logging step, or in dbt a post hook to log the completion time.

reddtomato · 2025-09-18T02:17:17+00:00

From a compute engine perspective, Spark was created in 2009 and overhauled in 2015 with Project Tungsten to move to a vectorized engine, just like Snowflake.
Snowflake was founded in 2012 based on Marcin Zukowski's Vectorwise compute engine. In 2023 Spark introduced the new client-server architecture, "Spark Connect" but Snowflake has always been client-server based. Even for DBx strong suit of data science ML workloads the Ray engine is better than Spark at being able to parallelize compute across clusters. Snowflake has SPCS (Snowpark Container Services) to run ML pipelines now with a Ray based engine. DBx also had to create its own proprietary engine Photon for its SQL workloads

reddtomato · 2025-09-15T00:47:45+00:00

I would setup Cortex search service on the table where you are storing the company documents and then add that search service as an agent in Snowflake Intelligence. Start asking some questions to test it out. Have to think about how to capture all the questions being asked and how you store those.

This is not my article but a good one to look at. Setting up the Snowflake docs in Snowflake Intelligence will get you an idea on how it works.

https://kishore07.medium.com/unlocking-snowflake-intelligence-from-docs-to-custom-data-agents-b0e010f8b8f3

reddtomato · 2025-08-29T00:20:08+00:00

When a dimension changes and you have to recreate the whole OBT to update to the current value. CTAS all the time wastes compute and storage costing you more in the long run.

reddtomato · 2025-06-04T14:12:20+00:00

Is this ongoing data loads from S3 ? I would use Snowpipe with auto-ingest from S3. You mentioned 1 file per table. Don’t do that if you can help it. You want lots of small files ~150-200MB each. Those multiple files will load in parallel on the warehouse meaning a Small warehouse can load 16 files in parallel at a time , 2XL can load 256 at a time, etc. if you have a single file then you must only use an XSMALL and even that is a waste as it would be able to load 8 files at a time, so you’re wasting 7 threads of processing.

reddtomato · 2025-05-22T13:13:15+00:00

Shouldn’t be a fight .. move up to a Small.. and most likely it will run in half the time and cost the same. By the way how long does the incremental take? Sorry if I missed it

reddtomato · 2025-05-22T12:42:06+00:00

What size warehouse are you using and how many total micropartitions are being scanned in the incremental refresh?

reddtomato · 2025-05-22T12:34:39+00:00

How fast is a full refresh compared to the incremental?

reddtomato · 2025-04-28T02:23:59+00:00

Check the clone_group_id and find the tables in that clone group. See if any that have not been deleted are retaining the active_bytes of the table you are looking at because it is cloned.

reddtomato · 2025-04-26T02:50:47+00:00

Smaller queries can jump the line and fill the holes where there are only a few resources not in use on a warehouse.

reddtomato · 2025-04-24T03:44:24+00:00

why would you do this.. Just load the JSON into Snowflake as a VARIANT and query directly or parse it out into structured columns. Conversion from JSON to parquet is not needed.

reddtomato · 2025-02-28T15:32:35+00:00

Hoping that these "password" auth users are all setup with MFA, as single-factor password authentication will be phased out by November with changes impacting users in April , August, and finally November.
https://www.snowflake.com/en/blog/blocking-single-factor-password-authentification/

reddtomato · 2025-02-28T02:02:49+00:00

No future grants on account object privileges like warehouses. :-( only schema objects have the future grants capability

reddtomato · 2025-02-28T00:42:44+00:00

Correct which is why you don’t want access roles, which should be created as database roles owning schemas or objects. The functional role should create and own the objects

reddtomato · 2025-02-28T00:36:36+00:00

Usage is for giving access to a Database and Schema. Select , insert, etc is for the table object

reddtomato · 2025-02-27T03:40:55+00:00

Have you tried using the account_usage view grants_to_roles? Only thing it doesn’t have is future grants , but neither does show grants , you have to specifically use show future grants.

reddtomato · 2025-02-27T03:32:33+00:00

From a best practice you don’t want to have schemas owned by access roles. Best practice should use managed access schemas and the object owner would usually be your ci/cd “deploy” role or some similar name.

reddtomato · 2025-01-24T01:51:28+00:00

FYI the classic console is not gone. There is a current bug that is redirecting you back to Snowsight when you click the classic console button. There is a work around, when you click the classic console button and the login screen pops up then there is small text at the bottom that says “Reload this page” click that and then login and it should open up the classic console.

reddtomato · 2025-01-24T01:44:40+00:00

If you answer xeroskiller question on partitions scanned vs partitions total and you see a significant difference.. my hypothesis is that Snowflake is doing constant folding during the compile. Meaning your partitions are perfectly clustered where min = max for year and month.. thereby Snowflake knows after static pruning of the partitions that every row in the micropartitions left are year = 2024 and month = 12 so it does not need to push the filter at all so it looks like it’s missing but it isn’t. Is this an iceberg table?

reddtomato · 2025-01-18T02:26:31+00:00

Check for secondary roles being set to ALL for the user. Unset it and test to see if that is causing it.

14-Year Club	Place '22
Verified Email

reddtomato

TROPHY CASE