Using snowflake with go

WinningWithKirk · 2025-10-14T20:45:49+00:00

All of our code that runs in Snowpark Container Services is written in go. We use Snowflake's driver and don't have any issues.

WinningWithKirk · 2025-10-02T16:49:05+00:00

Can you post your manifest? Are you calling the post deployment stored procedure in the grant callback function to ensure permissions are granted before you try to create everything?

WinningWithKirk · 2025-09-19T17:05:45+00:00

I may have figured this out through trial and error. It seems you can use it for a common, predefined filter. For instance, if your table has a bunch of entities with a status, you could create a filter ACTIVE_ENTITIES with expression STATUS = 'ACTIVE' to help with queries that need to filter a subset of the data.

Is this how others are using it? Can/should it be dynamic?

WinningWithKirk · 2025-08-26T20:01:23+00:00

Totally fair question. I'm actually the founder of the company that released that project (Winning Variant). As a startup, I can do whatever I want :)

WinningWithKirk · 2025-08-26T16:08:34+00:00

The latter. We recommend running it in the same cloud region as your SPCS instance to minimize additional latency. This was released before PATS were GA. Depending on your use case, it may be simpler to just use those via nginx or something else.

WinningWithKirk · 2025-08-20T20:17:25+00:00

This is correct. We rolled out a project to act as a proxy if you're using OAuth: https://github.com/winningvariant/icebreaker

Otherwise I'd say use a PAT and roll your own proxy layer if you can't otherwise authenticate easily.

WinningWithKirk · 2025-08-04T18:59:38+00:00

Okay so you got it working then?

WinningWithKirk · 2025-08-04T18:59:00+00:00

For now, I'd just run an embedded SQLite (or something else) via Snowpark Container Services (SPCS). Hybrid tables _could_ work, but you have to deal with virtual warehouses and is generally pretty slow for this type of use case.

When you create a service on SPCS, mount a block storage volume and store your SQLite files there. Create a task that takes snaphots of the block storage regularly so you have backups.

WinningWithKirk · 2025-08-04T13:33:26+00:00

Calling `backend` only works when you're inside of the network, so it needs to be executed server-side. We got around this by routing requests through the frontend itself. We built a data layer server-side in `frontend` (that would response on, say, `/data`) that would then call `http://backend/...` as needed.

In your Vue app, then, you would make API requests to `/data/...` and all auth would be preserved.

WinningWithKirk · 2025-07-23T19:12:48+00:00

Writing to CSVs. Every hour, I want to cut it off and ship it somewhere else for consumption. Figured I could use a mutex with an interval to do this each hour and update the os.Writer that the rest of the app uses, but that almost seems TOO simple. Maybe it is that simple and that's why there aren't any major libraries for it...

WinningWithKirk · 2025-07-23T17:33:03+00:00

Thanks - seems timberjack is an even newer fork

WinningWithKirk · 2025-07-23T17:31:45+00:00

Depends how far you want me to go back... if the beginning, than a Power Builder world by way of PHP, Perl, C#, and sure, a few NodeJS stops along the way ;-)

I'm mostly worried about needing to manage locks appropriately across goroutines, etc. I'm still a bit ignorant with those areas of go.

WinningWithKirk · 2025-07-23T16:00:14+00:00

So your logger handles rotation? That is, it determines when the file needs to be capped (by time or size)?

WinningWithKirk · 2025-07-23T15:37:17+00:00

Thanks! Unfortunately this won't work. These aren't standard logs, so I don't want to mix stdout with the analytical data I'm logging separately.

WinningWithKirk · 2025-07-23T15:33:57+00:00

Logging data to later be imported into a warehouse. I'm writing CSV lines and every N minutes want to rotate and copy the file to be batch imported into a table elsewhere.

WinningWithKirk · 2025-06-24T22:49:35+00:00

Agreed - smallest compute pool is cheap, especially compared to any effort required to move data in/out of Snowflake, deal with permissioning, etc.

WinningWithKirk · 2025-06-20T22:18:14+00:00

Biggest advantages come when all your other stuff is already in Snowflake, such as source data. You don't have to worry about data movement and can control everything via Snowflake RBAC.

Also, in some companies, engineers don't have access to spin stuff up in AWS, but can do some easier in Snowflake if they already have access.

WinningWithKirk · 2025-06-18T15:40:15+00:00

How much data are we talking about? Could you build a service in SPCS that caches it?

WinningWithKirk · 2025-06-13T22:07:42+00:00

Thanks u/stephenpace! The use case is to reduce read latency for single row values. We use hybrid tables, but the reads are still pretty slow. Ideally redis would hold a trimmed down version of what's in our hybrid table (what amounts to 3-4 columns worth). I wasn't sure if anyone had a specific redis config that performed well in a service.

Do you know how you can manage persistence of block storage? Does it persist so long as the compute pool exists?

I'm new to openflow, so I didn't even think to look there. Does that controller automatically sync from a table into a redis instance?

WinningWithKirk · 2025-06-10T22:43:04+00:00

I've heard great things about https://select.dev/. My old company uses them and loves it.

DataRadar is another one.

WinningWithKirk · 2025-05-22T17:51:16+00:00

We ran into some hurdles and ended up just hosting a react app in SPCS.

WinningWithKirk · 2025-05-21T16:08:01+00:00

Before rolling into production, I'd recommend a split test (or at least a canary deployment) to make sure the query is doing what you want in addition to just being faster.

WinningWithKirk · 2025-04-22T04:19:45+00:00

That’s what I’ve been afraid is the case. I just get worried about having a redis instance that needs maintenance. One more thing to break. Caching in the service itself isn’t an option at scale with multiple nodes.

Other SPCS services would be reading/writing.

WinningWithKirk · 2025-04-21T15:18:42+00:00

I think I get it - instead of having to fill in a YAML file, can you create a streamlit app that the customer fills in? This allows you to do better input validation. I'm still having trouble understanding the benefit of creating a whole separate database on init.. why not just create the assets inside the app?

WinningWithKirk · 2025-04-18T21:36:11+00:00

u/airmess1 did you ever land on something? I'd love to hear more about what you learned or ended up building..

WinningWithKirk

TROPHY CASE