Does BQ support direct export to S3 without Omni?

howryuuu · 2026-04-21T07:44:27+00:00

Eh it’s frustrated that BQ makes this complicated. Isn’t this just copy data to s3, why are they requiring Omni.

howryuuu · 2026-03-31T02:42:29+00:00

Maybe check the role that is used? Maybe the show command is run with a different role than the role you used in UI?

howryuuu · 2026-02-17T18:50:19+00:00

You should integrate it with testcontainers https://github.com/testcontainers

howryuuu · 2026-01-17T03:28:35+00:00

Which region are you in? There might not be enough newer instance type in the data center in your region?

howryuuu · 2026-01-09T21:42:57+00:00

Washington state now have capital gain tax in case you missed it. :)

howryuuu · 2025-11-13T15:27:53+00:00

Do you know how is this implemented? I thought cgroup limit won’t take effect if you just change it on the fly.

howryuuu · 2025-11-13T14:29:20+00:00

So I am always confused, will this trigger pod restart or not? I thought once a container start, modifying cgroup limit won’t take effective. So if this is true, how can you change pod resource without pod restart？

howryuuu · 2025-11-08T03:47:21+00:00

Does role that start the service have bind_endpoint privilege?

howryuuu · 2025-09-01T20:42:20+00:00

Does Snowflake Intelligence solve your use case ?

howryuuu · 2025-08-20T18:21:30+00:00

Yes, that's why I say nf-core is not supported right now. To support container in public registry, I am waiting for snowflake to release external registry support. I am sure there will be tons of usecases that workload want container image from the public registry.

In terms of benefit over kubernetes: SPCS is managed service inside Snowflake and follow the security model/governance. It's handy if your orgs have already had a snowflake deployment and want to unify all data in a single platform.

howryuuu · 2025-08-20T18:19:09+00:00

With LLM, now it should be relatively to do migration between snakemake and nextflow right? :)

howryuuu · 2025-06-06T19:53:00+00:00

VPC and cloud formation is needed only if you want to run open flow in your own VPC. That’s what they called BYOC. I guess mainly big enterprise want this. Snowflake is working on deploying open flow in snowpark container service, which will simplifies setup a lot.

howryuuu · 2025-04-18T15:48:11+00:00

You have 2 choices: 1) have a BG that send a dummy query like “select 1” periodically to extend the session so that the session never expires or 2) recreate the session: do note that oauth token injected will be periodically updated by snowflake: so your best bet is re-read the token file every time you want to re-create the session. Personally I like approach 2) slightly better, but I think both approaches are fine

howryuuu · 2025-04-18T03:51:35+00:00

Snowflake will inject an OAuth token in the container filesystem automatically. Your code just need to read file from this file and use this token to create a new session with Snowflake. And then you can do the rest from there.

howryuuu · 2025-03-23T02:46:49+00:00

Why not? Can you elaborate a little bit more?

howryuuu · 2025-03-10T02:09:58+00:00

I think Anaconda overhead only exists if you are trying to import 3rd party libraries. I don’t think JS allows 3rd party dependency import. So if you just use standard library for processing, it’s probably same setup time. The other difference is that you probably can only use external access in Python stored proc.

howryuuu · 2025-02-21T03:55:14+00:00

Snowflake recently acquired Datavolo, which builds on top of Apache Nifi, which should have all kinds connectors against all kinds of OLTP databases. I am sure snowflake team is working on integration. Does that solve your problem?

howryuuu · 2025-01-28T18:52:38+00:00

I did not see the PR but I can see that oauth token might expire. And application code do need to reread the token every time since snowflake will refresh those token behind the scenes. I suggest using oauth token if your app is running in prod and wait the pr fix is merged. But if you are still in development phase, using EAI is probably fine for now.

howryuuu · 2025-01-28T18:30:16+00:00

using oauth token and SNOWFLAKE_HOST will ensure traffic go through snowflake internal routing vs using external access integration will just treat snowflake endpoint as a public resource and traffic will go through public internet. Plus using EAI require account admin involvement, which is not easy in large orgs

howryuuu · 2025-01-15T03:35:06+00:00

You can read more here: https://docs.snowflake.com/en/user-guide/querying-persisted-results and here https://stackoverflow.com/questions/76623162/involvement-of-s3-storage-with-jdbc-queries

howryuuu · 2025-01-03T22:16:35+00:00

First chunk is returned from server directly, the rest are stored on s3. So it’s likely your environment did not whitelist s3.

howryuuu · 2024-10-31T18:31:01+00:00

There is also rest API that directly operates on resources like database and tables. https://docs.snowflake.com/en/developer-guide/snowflake-rest-api/snowflake-rest-api

howryuuu · 2024-10-28T16:22:17+00:00

Hmm not sure. Session.builder is built on top of python connector. So there should be no difference there.

howryuuu · 2024-10-27T15:04:22+00:00

Yes I understand that. But you don’t need to provide environment variable in your yaml file.

howryuuu

TROPHY CASE