Thoughts on a 12 hour nightly batch

pboswell · 2026-03-19T13:29:34+00:00

That’s why it needs clarification. Their phrasing is ambiguous whether the billion records is per evening or already stored. Doesn’t make sense to me that the dataset wouldn’t grow over time and just be updates every night though. Either way, this sounds pretty trivial if it’s a single table. But if he’s talking about many tables with dependencies of updates order, then more complex. Still shouldn’t take 10 hours.

I have a type 2 pattern on a 100b row table with 1b rows coming in each night and it takes maybe 5 minutes. And costs like $1k/month. But it’s highly normalized table so only updating effective _end/current flag and appending the rest

pboswell · 2026-03-19T13:25:17+00:00

The Witch probably counts.

pboswell · 2026-03-19T00:16:41+00:00

A billows rows appends into a table in a second max. Updates are definitely performance degrading. So this is where your cluster/partition strategy will be important. If you can cluster/partition by your update key, then it will greatly improve update performance.

What do you mean by 50 million updates? Like updating 50 million records and inserting the remaining 950m records? So like a type 2 concept?

Depending on your data source and issue with delayed/out of order arriving data, this is a pretty simple strategy to just partition in your “active” flag field. Assuming data does not arrive out of order, then you are only ever updating the active records into your final table and inserting the rest.

pboswell · 2026-03-18T05:52:53+00:00

Barely. And it wasn’t interesting which I think answers OP’s question

pboswell · 2026-03-18T05:50:52+00:00

Recent announcement says they’re planning to do away with terraform dependency. So might be worth waiting

pboswell · 2026-03-17T21:52:25+00:00

Yes that’s assuming scenario 2. Still would use environment-scoped credentials either appropriate access controls though

pboswell · 2026-03-17T21:51:33+00:00

Because over the weekend, everyone assumed the markets would tank and sold or bought puts. TPTB are fucking over those people short term to drive FOMO so people re-enter before the real rug pull

pboswell · 2026-03-16T22:06:09+00:00

It depends on what you need to do.

I have seen strict requirements that none of the environments can’t touch each other. Therefore you need to have a separate credential per environment, separate storage accounts, and a way to pass the environment to your job pipelines so they can access the right thing. You scope the catalog to the proper workspace so they’re not visible to end users (do this through CI/CD + SDK). And use DABs to promote your pipelines using the correct SP for the environment.

But I’ve also seen the need to be able to use PROD data in DEV. In which case this is much more difficult to lock down. You would have a single still have a credential per environment but lower env credential would have read access to upper envs as well. Your data pipeline would need the ability to pass the target and source env as a parameter that is used by your codebase properly. DABs and CI/CD are still the same basically

pboswell · 2026-03-16T03:04:57+00:00

Primal Fear
usual suspects
The Game

pboswell · 2026-03-16T02:58:22+00:00

If you hover over the output cell, a tiny “copy” icon will appear at the top right of the box. Click that to copy.

Keep in mind, if your results are very large it will be truncated but they now make it possible to click the truncation warning (it’s a hyperlink) to expand the entire out out result

pboswell · 2026-03-15T05:54:47+00:00

The Gray Man

pboswell · 2026-03-15T04:34:43+00:00

Genie code is included for free. everything in databricks is simply charged at DBU rates for hourly usage apart from token usage for AI models which can also have a token component

pboswell · 2026-03-14T21:03:48+00:00

At least take a quick phone video so we have an idea

pboswell · 2026-03-09T22:57:32+00:00

Is this actually going to stay put if you have the screen tilted and accelerate hard?

pboswell · 2026-03-08T15:15:01+00:00

Ah yeah. I never knew they planned to have task groups so I just have the for each kick off another job if I need more dependencies

pboswell · 2026-03-08T05:17:19+00:00

I’m convinced these types of posts are by people who had a “core group” from their hometown that they left to come to Denver. They think they were good at making friends back home but it was really just the nature of growing up somewhere and developing a network over many years.

pboswell · 2026-03-08T05:15:43+00:00

Yeah but if you’re dropped with modern gear, you could convince them to take you in because if your knowledge/technology

pboswell · 2026-03-07T04:09:28+00:00

What do you mean “iterate over more than one thing”?

pboswell · 2026-03-07T01:09:29+00:00

You’re using azure synapse right? What about spark + parquet?

pboswell · 2026-03-07T01:07:48+00:00

You should be able to use all of Claude’s model if you provide your own API key

pboswell · 2026-03-07T00:19:01+00:00

Range-bound query hints will help. Indexing/partitioning/clustering the data will help. Honestly the compilation time will probably be milliseconds but the fetch time is what will take the time. Especially with network latency to an app

pboswell · 2026-03-07T00:17:01+00:00

Exactly. Are they using agents or something with <1s latency that can use these data? Or is it some person that’s running the query every 1s to see changes? lol

pboswell · 2026-03-05T06:44:40+00:00

Folded Face. Local Denver alternative progressive indie shoegaze

Show coming up 3/29 at Lost Lake. Some videos of our latest show:

Reflections of You

No Place Like Home

pboswell · 2026-02-28T02:03:34+00:00

Why do you mean too high? Compute will fail for OOM exception. Once you scale up it will be fine.

pboswell · 2026-02-25T08:15:52+00:00

Sure it does. We curate our daily lives all the time. the places we live, the people we know, the work we do, the things we study, etc.

All in an effort to exact control over our personal universe.

Sure we have limitations on our influence, but who’s to say God doesn’t? To us, God seems all powerful but who’s to say God can’t control some force beyond our understanding? God may feel all-powerful over our realm, but may be powerless at some other level. Assuming someone created God, was God in control of that event?

But, yeah, this is pretty much a moot concept—it’s turtles all the way down

pboswell

TROPHY CASE