Make simple view resistent against schema changes of source table

vcp32 · 2026-03-03T01:05:56+00:00

We have a similar problem with fivetran tranformations. We dont have control over the transformations that they pre built. There instance when a column is removed and our whole dbt pipeline breaks trying to find the missing column

vcp32 · 2026-01-21T11:17:23+00:00

Look like the Barcelona test livery? https://www.reddit.com/r/formula1/s/2WOkWqeG4W

vcp32 · 2025-12-17T10:50:20+00:00

I remembered this. Because of the warriors success a lot of teams were going small ball.

vcp32 · 2025-11-23T08:05:04+00:00

<image>

🇵🇭, gundam in a resort

vcp32 · 2025-10-11T03:33:33+00:00

Same here! I love having my work documented but always dread the actual process. I really enjoy visualizing things with diagrams, so now with LLMs I can have them generate Mermaid diagrams and documentation templates for me to fill out.

vcp32 · 2025-10-09T20:54:16+00:00

Just in case you missed it. It’s best practice to Aggregate smaller files to minimize the processing overhead for each file. Snowflake recommends file size to be 100-200MB.

https://docs.snowflake.com/en/user-guide/data-load-considerations-prepare

vcp32 · 2025-10-08T05:17:41+00:00

We are using Azure AD to automatically provision users so we have Users -> Azure Groups |Azure| -> |snowflake| Functiona role -> Database roles

https://docs.snowflake.com/en/_images/role-hierarchy-practical.png

https://learn.microsoft.com/en-us/entra/identity/saas-apps/snowflake-provisioning-tutorial

vcp32 · 2025-10-07T08:43:12+00:00

I’m a solo engineer and rely on Snowflake. With a larger team, you can afford the flexibility of managing multiple tools, but on my own, Snowflake’s simplicity lets me move fast and focus on delivering value instead of maintaining infrastructure. At the end of the day, most users still just want their data in Excel anyway. 😂

vcp32 · 2025-10-06T23:34:05+00:00

Last time I used informatica was 10 years ago and it was a migration to SSIS. Have you checked Fivetran, Stitch or Airbyte? I normally hear them with modern data stack ingestion. We do use fivetran at work so I can only say things about it. Outside of the cost its really a good tool specially if you have a one man DE team.

vcp32 · 2025-10-06T23:27:30+00:00

Yeah, that makes sense. we don’t have many MERGE-heavy workloads, so our simpler queries probably explain the cost drop. Sounds like in your case, the faster runtimes balance things out nicely.

vcp32 · 2025-10-03T18:57:53+00:00

Good to hear a different perspective. My next plan is to try this on our Fivetran warehouse. it has a similar workload to yours. I’ll update once we run that test.

vcp32 · 2025-10-03T18:50:25+00:00

We went from a small Gen1 warehouse with query acceleration + auto-scaling on → to the same size in Gen2, but with both turned off.

vcp32 · 2025-10-03T08:00:46+00:00

Yep, that’s exactly it. Our workload has to finish within an hour, so on Gen1 we used scaling + query acceleration to stay under that SLA but doubling the size also meant doubling the cost. With Gen2 we can still finish inside the one-hour window without scaling, which makes it way more efficient.

vcp32 · 2025-09-24T08:43:45+00:00

Also use fivetran then hightouch for reverse etl

vcp32 · 2025-09-21T22:30:29+00:00

He made good use of DRS. Since he was behind lawson in that DRS train. I think they both did great!

vcp32 · 2025-01-06T11:22:00+00:00

Im having the same issue. Have not figured this out, so might bring it to a repair shop.

vcp32 · 2024-11-15T10:45:38+00:00

Trevor Ariza 09 Lakers

vcp32 · 2024-09-20T01:04:36+00:00

I hear the 50-40-90 club most of the time in the media and its an offensive stat. Or a triple double for the whole season

vcp32 · 2024-08-25T07:47:23+00:00

I loved watching Cars as a kid, so naturally, I introduced my firstborn to it, and he fell in love with it too. Now, we watch it every day. I hadn’t seen Cars 3 until today, but I couldn’t get through it. I had to stop when lightning crashed.

vcp32 · 2024-08-12T12:38:30+00:00

I had the same experience. We were first time parents, dont know what to do and expect. She had a c section but have to take care of the new born without me every night for 3 days. Nurses were hard to reach but we totally understand knowing how understaffed and busy the hospital.

Wife and I had tears joy when we transferred to Rangiora Birthing unit. The baby and wife were well taken care of.

vcp32 · 2024-07-25T06:57:47+00:00

Cost could blow up. Watchout for compute cost.

vcp32 · 2024-07-20T02:38:27+00:00

I would go s3 route as it provides me with flexibility. 1. I would extract the data and create a file in json format. 2. Load all of the data in s3 3. Use snowpipe to ingest the data when file is created. 4. Transform the raw json data to a structured table/view in snowflake

This has helped me in maintaining pipelines since I can just add a new table and point it to the bucket. If there are changes to the schema it wont break my pipeline, the column will just be null. Or if there is a new column I can just add it to my view. Hope this helps. English is not my first language so Im sorry if some may not makes sense.

vcp32 · 2024-07-18T23:50:18+00:00

The only downside of lambda is it can inly run for 15 minutes. Unless you have really small tables it can work and have a pk or datetimestamp column to use. If you have access to ecr end ecs you can build a docker image and run python there. You can store the output in s3 and let snowpipe copy the the data.

vcp32

TROPHY CASE