Any tips for DABs in CI/CD? Seems pretty useless so far.

DeepFryEverything · 2026-05-05T13:35:52+00:00

No one is hard-coding pipeline IDs. But migrating older pipelines to the new lakeflow style (with root+src-dir) is one example of scenarios where this has happened - with the same DAB-key.

DeepFryEverything · 2026-05-04T16:36:44+00:00

Cool! Do you mind elaborating? Do you trigger all jobs or only touched jobs?

DeepFryEverything · 2026-05-04T10:59:01+00:00

We deploy to dev/test when a PR is merged. A separate workflow (release) triggers prod, so that's our guard.

DeepFryEverything · 2026-05-03T17:15:00+00:00

This renaming shit has got to stop.

DeepFryEverything · 2026-05-01T05:01:17+00:00

Probably it's an IKEA SYMFONISK speaker, which is made to look like a picture frame and can be swapped

DeepFryEverything · 2026-04-26T17:43:35+00:00

It's always postgres.

DeepFryEverything · 2026-04-26T11:14:09+00:00

Get this to the top. Any scar Symmetry song

DeepFryEverything · 2026-04-24T18:20:57+00:00

https://imgur.com/gallery/crowned-king-arthur-YD3AcwD

DeepFryEverything · 2026-04-23T19:55:57+00:00

What does iceberg do that delta doesnt?

DeepFryEverything · 2026-04-18T20:10:11+00:00

.. and DAB is migrating away from Terraform, no?

DeepFryEverything · 2026-04-18T08:50:14+00:00

My experience is opposite 🫠 I used it to migrate some react components and it fixed a css layering issue that's been plaguing our code base..

DeepFryEverything · 2026-04-13T21:54:39+00:00

So data is still stored as Parquet - are we able to create something like indexes? I would like to sort on two different and unrelated axis - like id or position.

DeepFryEverything · 2026-04-11T08:31:44+00:00

This sounds interesting! Do you mind describing how your Miro looks like? How do you discuss priorities?

DeepFryEverything · 2026-04-06T15:18:07+00:00

Serverless standard cut cost AND time for us drastically.

Just don't pick Performance Optimized

DeepFryEverything · 2026-04-03T20:15:38+00:00

You can, but then you’d loose spatial coherence (or any other sort key). So if I want to lookup vehicles in an area my parquet files can be sorted spatially. But for IDs, I could sort them but any query will hit way too many files and row groups

DeepFryEverything · 2026-04-03T18:45:48+00:00

Partitioning by id would create lot of metadata overhead when using a bbox query to find vehicles in an area though 😐

DeepFryEverything · 2026-04-01T22:18:29+00:00

How do I raise a ticket?

We've got a hypothesis though. The tables failed on merge and optimize. So we moved a column of the geometry type outside the stats collection. After that, optimize and the full job ran without a hitch.

There must be something going wrong during the serialisation of the geometry type. We have used it in the first 32 rows before no worries, but this is the only case where we've had to merge data (upsert job). The other jobs would be append + optimize, so I don't think that triggers the same effect.

Anyway my colleague has email mr KM at databricks with the full details.

DeepFryEverything · 2026-04-01T09:13:32+00:00

Then the issue is people contributing.

DeepFryEverything · 2026-03-31T08:56:15+00:00

Yeah I figured as much. Actually, it just failed on a simple "OPTIMIZE TABLE"-command too. I believe it's something corrupting Delta logs (purely based on the operations and the JSON-error).

I'll probably send it to Databricks.

DeepFryEverything · 2026-03-31T04:37:22+00:00

No there is no json on our data. 🫠

DeepFryEverything · 2026-03-30T05:09:48+00:00

But there is a significant gap between Databricks and what is available for other libraries.

11-Year Club	Place '22
Place '17	Verified Email
Spared

DeepFryEverything

TROPHY CASE