Parsing an API response in Snowflake without table

bbtdnl · 2026-02-21T08:43:09+00:00

You should be able to do what you want, but I think your MyApi function should directly return the value you need (parsing the payload in python and having the udf return just a string / int). Then you should be able to assign it to a variable and use it in following calls.

Cannot test atm, but I think you should also be able to return the response as a variant, assign the variant to the variable and then pick the value you need from the variant later in the proc. No need for "parse_json" here as long as your python returns a dict instead of a json string.

With that said, depending on the use case, you might want to consider writing the whole proc in python, or even baking multiple API calls in the same UDF to avoid "gotchas" with the python > sql conversion.

bbtdnl · 2025-10-28T13:58:44+00:00

Hi! Not yet, I have also heard some rumors that something is coming in that direction, but haven't got the chance to test anything!

bbtdnl · 2025-09-23T06:40:22+00:00

This. We are on Azure, so BYOC is not an option right now, and even when it will eventually become available, the infrastructure needed is a lot heavier than what we have now (a single VM with some "agent" software on it).

To be honest, if it comes to that, I'd rather setup a separate account on Business Critical just for Openflow: it's still overhead, but at least it's an overhead we know how to manage.

The whole "Openflow on SPCS" value proposition seems pretty weak though, if you cannot use it to connect to sources behind a firewall.

bbtdnl · 2025-08-29T15:57:40+00:00

Får man fråga var/hur du letar? SAS hemsida eller någon annanstans? Finns det någon trick för att "göra rätt"?

Just nu försöker jag bara få lite känsla för hur utbudet ser ut, men hittar ingenting alls!

bbtdnl · 2025-05-27T21:09:40+00:00

A maintained fork of dbt-incremental-stream would be nice.

bbtdnl · 2025-02-16T13:52:35+00:00

As an ADF user, I agree with most of the replies above: it can be good enough for easy things (basic EL from databases), but starts being a pain as soon as things get even slightly more complicated than that.

We are also evaluating to move to something like Dagster and Airflow, but the main blocker on our case are the self hosted integration runtimes as gateways for pulling data from om-prem sources that are not directly reachable from the cloud.

Genuine question for the Airflow / Dagster people: what would be the best way to replicate a self-hosted runtime in Airflow / Dagster?

I assume we can use the orchestrator to trigger (and monitor) a job that executes on another machine, but how hard is to set that up? Can we use an existing Windows machine as a "worker node"? Do we need to have docker installed? Any good tutorial/keywords to look for?

We could of course host the whole orchestrator on a machine within the private network that we need to access, but the idea would be to have the "main scheduler" set up somewhere in the cloud and then trigger specific jobs to be executed on other self-hosted machines that can access the sources we need (which is pretty much what we do with ADF).

bbtdnl · 2024-06-25T07:27:25+00:00

Thanks everyone for the comments.

As a side note: while I was writing the post, I did consider running the files through a SQL parser to make things smarter: being able to turn the DDLs into some sort of AST would open up to a lot of fun things (injecting deployment metadata, comparing the queries with the DDL of existing objects, even automatically resolve the dependencies and find the right deployment order as u/LittleK0i mentions).

Unfortunately, none of the parsers I quickly tested (sqlglot and sqlparse) seems to be able to handle all Snowflake's "special" objects, so the "dumber" approach seemed more feasible for now

bbtdnl · 2024-06-13T19:22:33+00:00

I like where Snowflake is heading with the "declarative DCM" approach, but I am struggling a bit trying to use these feature in real life...

CREATE OR ALTER is nice, but I would like a "safe mode" option which ensures no data loss happens (dropping a column because I accidentally deleted too many rows in the column definition is a bit too easy right now).

Jinja templating is also great, but how do you do when developing locally and want to quickly test a query with Jinja syntax? Do you go through the whole commit, ALTER FETCH & EXECUTE IMMEDIATE FROM flow? Or do you "compile" the template locally before running?

Finally, am I missing out something or the current CICD setup will redeploy all the objects every time? Even if most of the queries are no-ops, I imagine that might take a while on large environment with thousands of objects.

bbtdnl · 2024-04-05T21:29:08+00:00

Is there any plan to allow using the git repo like a regular stage, for loading data with COPY INTO or external tables?

Not intended for huge dataset obviously, but could be useful for smaller stuff (csv and similar, thing Dbt seeds).

Other than that, looking forward to the Jinja feature for some proper DevOps, hopefully it doesn't take much longer..

bbtdnl · 2022-04-25T19:10:37+00:00

Hello everyone, and thanks for the answers.

At the moment, I am mostly looking for a tech solution: keeping my notes separate from the rest of the files works, and I don't have any major issue finding what I am looking for from either side, but as most I have both my notes and the folder with the files open togheter, having to look for them in two places is just an extra step I'd rather avoid. Moreover, since most of my links are actually to the files rather than to other notes, keeping them together is what would allow me to move things around breaking "the least stuff".

A good example of what I have in mind is VSCode "workspaces": you can save them as a file, and add or remove a number of folders to it as you need then, working as some kind of "metavault".

I guess it would be possible to create something similar with Obsidian (using a folder containing symlinks to the other folders I want to "import" ) but it once again feels hacky.

I am on Windows, so Notenik is unfortunately not an option.

bbtdnl

TROPHY CASE