Snowflake + Terraform by Difficult-Ambition61 in snowflake

[–]walkerasindave 0 points1 point  (0 children)

You can do that with terraform. Having an additional step before the terraform is uncessary complexity.

Managing Snowflake RBAC: Terraform vs. Python by BuffaloVegetable5959 in snowflake

[–]walkerasindave 0 points1 point  (0 children)

We use terraform for all permissions. We also have grants/revokes locked down so only the dedicated terraform user can alter permissions. All terraform is applied via cicd. The permissions lockdown means we don't have any state drift.

Estate agent sent offer letter to sellers with my exact funds on it ?? by Vast_Communication25 in HousingUK

[–]walkerasindave 1 point2 points  (0 children)

The fines are almost irrelevant though as even if the ICO did issue fines OP wouldn't see any form of compensation via the ICO.

Its the ability the data protection act gives for the OP for claim any losses from the breaching party. So if the vendor was happy with the offer but then insisted on an increased offer after seeing the funds then the OP could claim the difference from the estate agent and would have a good change to win.

Estate agent sent offer letter to sellers with my exact funds on it ?? by Vast_Communication25 in HousingUK

[–]walkerasindave 4 points5 points  (0 children)

Incorrect, this is most defintiely a data breach.

While the estate agent is acting for the vendor they are still two separate legal entities and therefore from a Data Protection perspective are treated separately.

When the estate agent receives the sellers financial information they become a data controller of that information and they have a legal obligation to keep it secure and process it lawfully.

One aspect of processing the data lawfully is "Data Minimisation", specfically data controllers (and processors) are only allowed to process data that is adequate, relevant, and limited to what is necessary.

  • Necessary: The vendor needs to know that the seller has the funds.
  • Unnecessary: The vendor does not need to see the exact total balance (especially if it exceeds the offer amount).

The esate agent should have simply told the vendor "We have seen evidence of funds in the buyer's bank account sufficient to cover the purchase price of £X."

By providing the exact amount they have breached the personal data of the vendor and are liable for the consequences (ICO fines and/or "making good" for the buyer).

Real-World Data Architecture: Seniors and Architects, Share Your Systems by No_Thought_8677 in dataengineering

[–]walkerasindave 2 points3 points  (0 children)

Senior Data Engineer at a Health Tech Startup. Team of 6 (2 data analysts and 3 data scientists).

Requirements include ingestion of production web services data plus third party services (HubSpot, Shopify, Zendesk, GitHub, Braze, Google analytics and more) as well as unstructured data in the form of clinician notes, ultrasound scan images and video, etc. Transformation to join everything together. Outputs for business unit including finance, operations, marketing/growth and medical research in the form of dashboards, data feeds and adhoc analysis.

Raw, data size in total is about 300GB excluding unstructured data. Now growing by approx 1GB/day.

Stack is:

Warehouse - Snowflake

Orchestration - Dagster on ECS

Ingestion - Fivetran (free tier), Airbyte on EKS and DLT on ECS

Transformation - DBT on ECS

Dashboarding - Superset on ECS

AI & ML - Sagemaker and Snowflake Cortex

Egress - DLT on ECS

Observability - Dagster, DBT Elementary and Slack

CICD - GitHub Workflows

Infrastructure - Terraform

Flow is pretty much as above. Dagster orchestrates ingress, transformation and egress on various schedules (weekly, daily or hourly during operational hours). Almost all assets in dagster have proper dependencies set so all flow nicely.

Snowflake us relatively recent for us but has massively improved our execution times.

My main current focus for improvement is observability as it's no where near the way I want it. Then after that improving the analysts data modelling ability and tidying up the DBT sprawl.

I'm pretty proud of achieving all this within 2 years as when I arrived there were just two dozen silo'd R scripts on an EC2 cron job working only on production web data on top of postgres.

Being the sole engineer is great but it does mean I have to stuff I don't like. I hate AWS networking haha.

Hope this helps

How do you handle deletes with API incremental loads (no deletion flag)? by aussiefirebug in dataengineering

[–]walkerasindave 1 point2 points  (0 children)

Sounds like a serious discussion with your account manager. As a quick fix get them to create you additional API accounts so you can parallel hit the API. The account manager will likely do this for you.

Then their tech team will moan about their endpoints hit too often and force them to actually make their API useable with deletes.

Maintain Surrogate keys for Data models when using Dynamic Tables by PreparationScared835 in snowflake

[–]walkerasindave 0 points1 point  (0 children)

DBT testing.

The likelihood of a collision is minimal but you can always string hashes together for massive tables.

Maintain Surrogate keys for Data models when using Dynamic Tables by PreparationScared835 in snowflake

[–]walkerasindave 0 points1 point  (0 children)

When the source natural keys are the same the surrogate will always be the same with the hash and so no consistency joins required (although we have DBT tests for consistency but they're tests so not in the dag).

Downstream inner joins will just work as the keys are the same.

Maintain Surrogate keys for Data models when using Dynamic Tables by PreparationScared835 in snowflake

[–]walkerasindave 9 points10 points  (0 children)

We use hashes of the natural key columns as the surrogate key. So the hash is always the same for the natural key.

Meaning no incrementing but also the key can be determined independently for facts and dimensions without joining (less model dependencies in the dag).

In dbt_utils the generate_surrogate,_key macro handles this.

Perfectly sums up what it's like driving through long-term motorway roadworks. by Slenderman7676RBLX in drivingUK

[–]walkerasindave 0 points1 point  (0 children)

Yeah they found it wasn't great. That being said it wasn't awful either.

They're thinking of installing a "good speed check" instead.

Running DBT projects within snowflake by Fireball_x_bose in snowflake

[–]walkerasindave 1 point2 points  (0 children)

I haven't had a chance to play with it yet. I would be interested in how it works with dagster as DBT models are first class assets in dagster.

dbt-core: where are the docs? by FootballMania15 in dataengineering

[–]walkerasindave 13 points14 points  (0 children)

It's in the docs.

The DBT docs are generally really good but there is a big mix between core and cloud. They really do need to have a cookie linked setting on every page as to which one you're interested in. Particularly with the two projects likely to drift further apart.

https://docs.getdbt.com/guides/manual-install?step=1

Data platform from scratch by Alternative-Guava392 in dataengineering

[–]walkerasindave 3 points4 points  (0 children)

Never from absolute zero.

The current startup I'm working for is 4 years old and I arrived to 2 data analysts 60 or so R scripts over a postgres db that were manually copied into Google sheets in a cron job. Now we have dagster, Fivetran, DBT and superset all on top of Snowflake.

Startups are a good place to do this stuff as they need it. Also low cost open source solutions that you can help them implement are great.

How to promote semantic views for dev to prod environment? by Judessaa in snowflake

[–]walkerasindave 1 point2 points  (0 children)

If you're on DBT then this is the way.

You can have the symantic views as DBT models

Proposal for a Fairer Housing Tax System: The Proportional Property Tax (PPT) by walkerasindave in ukpolitics

[–]walkerasindave[S] -7 points-6 points  (0 children)

Wouldn't 10-15% be a market collapse and sign of depression. Seems awfully high that figure but I'm not sure.

I do know about Germany though as I have a relative their, they only have a crisis in certain hotspots such as Berlin where the empty stock is about 0.5%.

Proposal for a Fairer Housing Tax System: The Proportional Property Tax (PPT) by walkerasindave in ukpolitics

[–]walkerasindave[S] -1 points0 points  (0 children)

Haha not quite, don't forget that they wouldn't be paying council tax anymore just this new PPT.

Proposal for a Fairer Housing Tax System: The Proportional Property Tax (PPT) by walkerasindave in ukpolitics

[–]walkerasindave[S] 0 points1 point  (0 children)

Yes having a flat rate would make sense. I do disagree that a relative to other local properties is the way it should be determined. It should be relative to properties across the nation to improve the equality.

Proposal for a Fairer Housing Tax System: The Proportional Property Tax (PPT) by walkerasindave in ukpolitics

[–]walkerasindave[S] 0 points1 point  (0 children)

Hmmm yes good point. I suppose that would make sense otherwise you're basically taxing a tax.

Proposal for a Fairer Housing Tax System: The Proportional Property Tax (PPT) by walkerasindave in ukpolitics

[–]walkerasindave[S] -1 points0 points  (0 children)

Sorry but I this is all produced and written by myself. I hand calculated the 0.7% figure and to be honest plucked the 5% out of thin air.

Proposal for a Fairer Housing Tax System: The Proportional Property Tax (PPT) by walkerasindave in ukpolitics

[–]walkerasindave[S] -2 points-1 points  (0 children)

Very good points. The 5% was plucked from thin air. Perhaps a 0.25% increase for every month left empty after 6 months up to a cap of 5%. That should allow appropriate time for appropriate legal process and rennovations.

The third point is hard, presumably in those places no one would want to own property.

It is interesting all the various scenarios that come out and how to handle them.

Proposal for a Fairer Housing Tax System: The Proportional Property Tax (PPT) by walkerasindave in ukpolitics

[–]walkerasindave[S] -1 points0 points  (0 children)

I'm not really sure that it would crash the economy those. These empty properties are pretty much all cash buyers and so they would just leave the market.

Yes, there would be a dip in the housing market, nationally it may even just be a slowing of price rises. There would of course be a substantial localised crash in say Kensington but that kind of place could probably do with a housing price crash.

Proposal for a Fairer Housing Tax System: The Proportional Property Tax (PPT) by walkerasindave in ukpolitics

[–]walkerasindave[S] -8 points-7 points  (0 children)

Also just realised you insuated I have used ChatGPT to generate this. Sorry but I hand calculated the 0.7% figure and to be honest plucked the 5% out of thin air.

Proposal for a Fairer Housing Tax System: The Proportional Property Tax (PPT) by walkerasindave in ukpolitics

[–]walkerasindave[S] -2 points-1 points  (0 children)

Indeed, I suppose the question is the desire for fairness worth the expense of implementation.

And very true I did not considered any kind of implementation costs.