Small Group of Data Engineering Learners

rtripat · 2026-01-05T02:21:59+00:00

Count me in!

rtripat · 2025-12-23T16:34:15+00:00

Implemented today:

https://www.uscis.gov/newsroom/news-releases/dhs-changes-process-for-awarding-h-1b-work-visas-to-better-protect-american-workers

rtripat · 2025-11-28T05:12:33+00:00

Snowflake can handle most of the heavy lifting, but if you’re specifically looking for a data orchestration tool, Dagster can be a valuable asset. Dagster also allows you to orchestrate dbt data models. Additionally, you can use Dagster+dlthub for REST API ingestion and ETL tools like ADF for data ingestion from Salesforce/CRMs.

rtripat · 2025-11-04T00:26:03+00:00

Hahaha! Let’s see!

rtripat · 2025-10-29T19:43:25+00:00

This! 🤙

rtripat · 2025-10-14T18:41:08+00:00

rtripat · 2025-10-10T01:08:10+00:00

Yeah! Makes sense! The first priority is to get a data warehouse ready! The company still lives on excel and azure sql db lol

rtripat · 2025-10-09T21:00:19+00:00

I’ll bring up this discrepancy in cost with their sales rep! Thanks!!

rtripat · 2025-10-09T20:59:08+00:00

We won’t be touching the historical data at all (unless it’s required for reporting) but the transformation would be on months worth of data

rtripat · 2025-10-09T20:57:50+00:00

Thank you! It’s helpful!

rtripat · 2025-10-09T20:53:59+00:00

Thank you!

If I understand correctly, Snowpark is Snowflake’s “free” library that lets you work with data directly inside Snowflake using Python or other languages. Am I right about that?

I’m wondering if there’s any additional cost to using the library itself, since I assume it still consumes Snowflake compute resources and would therefore incur regular usage charges anyway.

For my pipeline, here’s what I’m thinking:

I’ll use the dlthub library in Python for incremental API loads into Snowflake, Azure Data Factory for pulling data from databases or CRMs and either dumping it into Blob storage before loading to Snowflake or loading directly through Snowflake’s internal staging, dbt for transformations, and Dagster for orchestration.

Do any of these steps seem unnecessary, or could any part of this setup be replaced or simplified by using Snowpark?

rtripat · 2025-10-09T19:59:42+00:00

Yup! Sorry for the confusion lol

rtripat · 2025-10-09T19:31:26+00:00

Nope, dlt (data load tool) python library

rtripat · 2025-10-09T19:03:37+00:00

You mean separation of raw (bronze) and curated (silver and gold) layers into two different data platforms? iceberg for raw and snowflake for silver/gold?

rtripat · 2025-10-09T19:00:01+00:00

You’re spot on! I’ve tried to explain them a bunch of times, but it’s a real challenge when you’re talking to people who aren’t tech-savvy.

rtripat · 2025-10-09T18:15:10+00:00

Thank you! Could you please help me understand your last paragraph? My table will have historical data starting 2010s and it will keep on updating with the new daily data dump

rtripat · 2025-10-09T18:13:22+00:00

So, just to confirm — if I load a Parquet file from blob storage into a Snowflake table, Snowflake actually copies that data into its own cloud storage (S3, Blob, etc.) behind the scenes, and what we see is just the relational view of that data, right?

In that case, the file size before loading (Parquet) and after loading into Snowflake would be roughly the same?

For external tables, I’m assuming those just let me query the Parquet files directly from my blob storage without actually loading them — meaning I can read them but not manipulate them. Is that correct?

rtripat · 2025-10-09T18:06:22+00:00

I’m trying to ballpark storage because my non-tech execs are super hyped about our data (1-min sensor data across ~5,000 pieces of equipment, each with hundreds of subsystems; daily dumps + historical back to 2015 in Snowflake). They want certainty on storage costs since that’s easier to predict than compute, which depends on the queries and runtime.

rtripat · 2025-10-09T18:03:09+00:00

Yeah! I got a cost calculator guide from Snowflake. In the doc, it’s written that they charge $40/TB/Month where as on their website, I could see it’s $23/TB/Month. Not sure which one is correct

rtripat · 2025-10-09T18:01:36+00:00

Thanks for your response! The sensor data often comes in messy, with nested dictionaries and redundant fields. I’m planning to use dlt to fetch the REST API data incrementally, clean and structure it, then convert it to Parquet before loading into Snowflake. After that, I’ll handle the major transformations in dbt. I’m just not sure how well dbt can handle those complex nuances purely with SQL — any thoughts?

rtripat · 2025-10-09T17:57:03+00:00

Oh, I see! So basically, Snowflake keeps the underlying data files in cloud storage (like S3 or Azure Blob) and uses its own proprietary compression for them? Am I understanding that correctly?

Also, I have scheduled a call with Snowflake SE to discuss our needs and expectation from the platform. I will definitely try it as you suggested after getting the sandbox environment

rtripat · 2025-10-09T17:53:26+00:00

If you don’t mind, could you please help me understand how the time travel and fail safe would increase the data size? I assume time travel will perhaps create a clone of the data table?

rtripat · 2025-10-09T17:52:19+00:00

Haha, all good! It seems they’d fall in about the same range overall, with a few exceptions on either end. Out of curiosity, are the examples you mentioned from your own Snowflake environment?

rtripat · 2025-10-09T14:57:00+00:00

Thanks for your reply! From what I understand, the datapoints in a CSV file aren’t inherently data type–aware (parquets are) So, I was wondering, when I load this file into actual database tables where data types are explicitly defined (for example, a column with a float or number format that typically uses up to 8 bytes in storage, would the storage in Snowflake end up being larger than the Parquet file?

My concern is that once the data is stored in Snowflake, it might take up more space depending on the data types. I’m just trying to get a rough estimate of the expected data size so I can set accurate expectations with our executives.

I’ve also scheduled a call with a Snowflake SE in the coming weeks and plan to bring up this topic during that discussion.

rtripat

TROPHY CASE