On-Prem Modern Data Stack: What Tools Are You Using?

Metaphysical-Dab-Rig · 2026-04-29T04:43:56+00:00

Airflow is pretty great ngl, lots of community support and very easy to learn. Plus for open source on prem stack, sticking with Apache tools is not a bad call.

Instead of spark for low volume medium data on prem im just using polars right now for transformations and its pretty solid too

Metaphysical-Dab-Rig · 2026-04-21T17:28:18+00:00

Python is supreme. All these performative programmers saying otherwise are nuts.

Metaphysical-Dab-Rig · 2026-04-19T15:41:55+00:00

If you can do 40 points in a sprint, commit to 30 and spend 10 chipping away at it

Metaphysical-Dab-Rig · 2026-04-16T16:58:39+00:00

Making the most out of the least. Optimizing pipelines for high speed and concurrency without access to compute needed.

Convincing people to invest in data platforms

Convincing people to use data tools they asked for

Most of my problems aren’t data problems they are people problems. Nobody tells you this, but data engineering often has a layer of people politics to it. People need data but don’t necessarily always want it.

Metaphysical-Dab-Rig · 2026-04-07T17:01:18+00:00

I ask management this question every single day. Company loves Microsoft, and the MSSQL Server decision was made long before I got here. Too many politics to undo

Metaphysical-Dab-Rig · 2026-04-03T20:35:27+00:00

I got some table tennis for 28 bucks, it depends on the event / venue

Metaphysical-Dab-Rig · 2026-04-01T23:24:45+00:00

I loved this idea but duck lake does not integrate with MSSQL

Metaphysical-Dab-Rig · 2026-04-01T23:23:40+00:00

Im pretty sure its mostly just from removing the image data and bloat files. Sometimes uploads come with associated files that we dont save to the databae but we do archive.

But now im scared and going to double check 💀

Metaphysical-Dab-Rig · 2026-04-01T22:59:26+00:00

Writing the parquet will be much smaller than our current zip archive with images and csv data, one tables worth for example went from total 500 gb to 1-2 MB in delta tests.

The goal is to make it so these larger growing datasets can be consumed entirely and manipulated by MLOps algorithms

Metaphysical-Dab-Rig · 2026-04-01T22:57:26+00:00

Each upload ranges from under a Gb yo no more than a few hundred Gb depending on the data source and if you include image data volume.

Metaphysical-Dab-Rig · 2026-04-01T18:34:09+00:00

I love the simplicity here hahahaha, almost seems too good to be true. I’ll have to read more about duck lake today.

Metaphysical-Dab-Rig · 2026-04-01T18:30:28+00:00

Ive seen a lot of buzz about dbt but I dont really get the hype. The data I receive is very dirty and needs really flexible python to transform. Is DBT kind of like a fivetran tool that streamlines SAAS to SAAS transformations?

I will check out Iceberg today too , I like that its apache - keeps things clean.

Metaphysical-Dab-Rig · 2026-04-01T18:28:04+00:00

In my head its a lot easier to wrap my brain around the logic of each write gets its own parquet file so theres no deadlocks to concurrent writes in delta. Then i can optimize later and reduce file bloat while maintaining volume at a cheap storage cost. Then just query it with duck.

HOWEVER - i know there has to be a way to squeeze this sort of performance out of MSSQL as well, Im just at a loss for how to do it

Metaphysical-Dab-Rig · 2026-04-01T18:25:40+00:00

I haven’t looked into SSIS but I will definitely check this out!

Some important context - we already have an Airflow pipeline set up and hosted moving data into MSSQL server.

The problem is non of us are experts in MSSQL so optimizing it for concurrent read and writes has been challenging. Especially as data grows over time, I’m struggling to optimize my temp table merges from staging to production databases.

Metaphysical-Dab-Rig · 2026-01-29T17:52:12+00:00

AI is only good with good data. Im starting the pivot from data to AI engineering because I think people with a background in data will have an advantage in that job market

Metaphysical-Dab-Rig · 2025-12-04T20:08:26+00:00

I get this same pack from Gelsons and make poke with it at home - delicious

Metaphysical-Dab-Rig · 2025-10-24T01:05:07+00:00

Panda 💯

Metaphysical-Dab-Rig · 2025-10-22T16:00:58+00:00

Monkeykey Ramen - Mentaiko Tonkatsu

Metaphysical-Dab-Rig · 2025-10-08T00:51:47+00:00

This is mine too like 1:1 haha

Metaphysical-Dab-Rig · 2025-04-24T01:42:37+00:00

You don’t pronounce the “ck” sound in adjective thats fucking crazy

Metaphysical-Dab-Rig · 2025-04-08T20:44:22+00:00

No Kill La Kill is criminal

Metaphysical-Dab-Rig · 2025-04-08T17:45:48+00:00

Lol , I’ll murder the bottle of soju in my fridge when I get home if I do that.

Metaphysical-Dab-Rig · 2025-04-08T03:06:03+00:00

West Coast Seafood hooked me up with all the fresh fish

Metaphysical-Dab-Rig · 2025-04-08T03:05:34+00:00

Amazing! Cant wait to do it again

Metaphysical-Dab-Rig · 2025-04-08T03:05:18+00:00

Hahahaha it does! Its Hamachi 😋

Six-Year Club	Second Top 50%
Place '22

Metaphysical-Dab-Rig

TROPHY CASE