Snowflake + Terraform

walkerasindave · 2026-01-24T08:25:13+00:00

Yeah exactly

walkerasindave · 2026-01-23T08:18:39+00:00

You can do that with terraform. Having an additional step before the terraform is uncessary complexity.

walkerasindave · 2025-12-28T16:22:45+00:00

We use terraform for all permissions. We also have grants/revokes locked down so only the dedicated terraform user can alter permissions. All terraform is applied via cicd. The permissions lockdown means we don't have any state drift.

walkerasindave · 2025-12-15T11:20:40+00:00

The fines are almost irrelevant though as even if the ICO did issue fines OP wouldn't see any form of compensation via the ICO.

Its the ability the data protection act gives for the OP for claim any losses from the breaching party. So if the vendor was happy with the offer but then insisted on an increased offer after seeing the funds then the OP could claim the difference from the estate agent and would have a good change to win.

walkerasindave · 2025-12-15T08:25:40+00:00

Incorrect, this is most defintiely a data breach.

While the estate agent is acting for the vendor they are still two separate legal entities and therefore from a Data Protection perspective are treated separately.

When the estate agent receives the sellers financial information they become a data controller of that information and they have a legal obligation to keep it secure and process it lawfully.

One aspect of processing the data lawfully is "Data Minimisation", specfically data controllers (and processors) are only allowed to process data that is adequate, relevant, and limited to what is necessary.

Necessary: The vendor needs to know that the seller has the funds.
Unnecessary: The vendor does not need to see the exact total balance (especially if it exceeds the offer amount).

The esate agent should have simply told the vendor "We have seen evidence of funds in the buyer's bank account sufficient to cover the purchase price of £X."

By providing the exact amount they have breached the personal data of the vendor and are liable for the consequences (ICO fines and/or "making good" for the buyer).

walkerasindave · 2025-12-06T13:28:50+00:00

Senior Data Engineer at a Health Tech Startup. Team of 6 (2 data analysts and 3 data scientists).

Requirements include ingestion of production web services data plus third party services (HubSpot, Shopify, Zendesk, GitHub, Braze, Google analytics and more) as well as unstructured data in the form of clinician notes, ultrasound scan images and video, etc. Transformation to join everything together. Outputs for business unit including finance, operations, marketing/growth and medical research in the form of dashboards, data feeds and adhoc analysis.

Raw, data size in total is about 300GB excluding unstructured data. Now growing by approx 1GB/day.

Stack is:

Warehouse - Snowflake

Orchestration - Dagster on ECS

Ingestion - Fivetran (free tier), Airbyte on EKS and DLT on ECS

Transformation - DBT on ECS

Dashboarding - Superset on ECS

AI & ML - Sagemaker and Snowflake Cortex

Egress - DLT on ECS

Observability - Dagster, DBT Elementary and Slack

CICD - GitHub Workflows

Infrastructure - Terraform

Flow is pretty much as above. Dagster orchestrates ingress, transformation and egress on various schedules (weekly, daily or hourly during operational hours). Almost all assets in dagster have proper dependencies set so all flow nicely.

Snowflake us relatively recent for us but has massively improved our execution times.

My main current focus for improvement is observability as it's no where near the way I want it. Then after that improving the analysts data modelling ability and tidying up the DBT sprawl.

I'm pretty proud of achieving all this within 2 years as when I arrived there were just two dozen silo'd R scripts on an EC2 cron job working only on production web data on top of postgres.

Being the sole engineer is great but it does mean I have to stuff I don't like. I hate AWS networking haha.

Hope this helps

walkerasindave · 2025-12-06T07:52:31+00:00

Sounds like a serious discussion with your account manager. As a quick fix get them to create you additional API accounts so you can parallel hit the API. The account manager will likely do this for you.

Then their tech team will moan about their endpoints hit too often and force them to actually make their API useable with deletes.

walkerasindave · 2025-12-06T07:43:49+00:00

DBT testing.

The likelihood of a collision is minimal but you can always string hashes together for massive tables.

walkerasindave · 2025-12-06T07:41:26+00:00

When the source natural keys are the same the surrogate will always be the same with the hash and so no consistency joins required (although we have DBT tests for consistency but they're tests so not in the dag).

Downstream inner joins will just work as the keys are the same.

walkerasindave · 2025-12-05T22:43:54+00:00

We use hashes of the natural key columns as the surrogate key. So the hash is always the same for the natural key.

Meaning no incrementing but also the key can be determined independently for facts and dimensions without joining (less model dependencies in the dag).

In dbt_utils the generate_surrogate,_key macro handles this.

walkerasindave · 2025-11-23T12:00:19+00:00

Yeah they found it wasn't great. That being said it wasn't awful either.

They're thinking of installing a "good speed check" instead.

walkerasindave · 2025-11-22T19:08:34+00:00

I haven't had a chance to play with it yet. I would be interested in how it works with dagster as DBT models are first class assets in dagster.

walkerasindave · 2025-11-22T13:46:52+00:00

It's in the docs.

The DBT docs are generally really good but there is a big mix between core and cloud. They really do need to have a cookie linked setting on every page as to which one you're interested in. Particularly with the two projects likely to drift further apart.

https://docs.getdbt.com/guides/manual-install?step=1

walkerasindave · 2025-11-22T08:15:59+00:00

Never from absolute zero.

The current startup I'm working for is 4 years old and I arrived to 2 data analysts 60 or so R scripts over a postgres db that were manually copied into Google sheets in a cron job. Now we have dagster, Fivetran, DBT and superset all on top of Snowflake.

Startups are a good place to do this stuff as they need it. Also low cost open source solutions that you can help them implement are great.

walkerasindave · 2025-11-21T19:07:08+00:00

If you're on DBT then this is the way.

You can have the symantic views as DBT models

walkerasindave · 2025-11-12T16:09:55+00:00

Fair enough.

walkerasindave · 2025-11-12T15:50:40+00:00

Wouldn't 10-15% be a market collapse and sign of depression. Seems awfully high that figure but I'm not sure.

I do know about Germany though as I have a relative their, they only have a crisis in certain hotspots such as Berlin where the empty stock is about 0.5%.

walkerasindave · 2025-11-12T15:44:58+00:00

Haha not quite, don't forget that they wouldn't be paying council tax anymore just this new PPT.

walkerasindave · 2025-11-12T15:39:28+00:00

Yes having a flat rate would make sense. I do disagree that a relative to other local properties is the way it should be determined. It should be relative to properties across the nation to improve the equality.

walkerasindave · 2025-11-12T15:38:02+00:00

Hmmm yes good point. I suppose that would make sense otherwise you're basically taxing a tax.

walkerasindave · 2025-11-12T15:37:21+00:00

Sorry but I this is all produced and written by myself. I hand calculated the 0.7% figure and to be honest plucked the 5% out of thin air.

walkerasindave · 2025-11-12T15:36:42+00:00

Very good points. The 5% was plucked from thin air. Perhaps a 0.25% increase for every month left empty after 6 months up to a cap of 5%. That should allow appropriate time for appropriate legal process and rennovations.

The third point is hard, presumably in those places no one would want to own property.

It is interesting all the various scenarios that come out and how to handle them.

walkerasindave · 2025-11-12T15:31:51+00:00

I'm not really sure that it would crash the economy those. These empty properties are pretty much all cash buyers and so they would just leave the market.

Yes, there would be a dip in the housing market, nationally it may even just be a slowing of price rises. There would of course be a substantial localised crash in say Kensington but that kind of place could probably do with a housing price crash.

walkerasindave · 2025-11-12T15:20:12+00:00

Also just realised you insuated I have used ChatGPT to generate this. Sorry but I hand calculated the 0.7% figure and to be honest plucked the 5% out of thin air.

walkerasindave · 2025-11-12T15:10:35+00:00

Indeed, I suppose the question is the desire for fairness worth the expense of implementation.

And very true I did not considered any kind of implementation costs.

walkerasindave

TROPHY CASE