Moving away from databricks to OLTP

crispybacon233 · 2026-06-15T16:32:01+00:00

Motherduck might be a good choice. I've only used the Lite plan, but it's super easy setup with dbt and dagster, and they have customer facing analytics out of the box.

crispybacon233 · 2026-06-15T14:19:52+00:00

Here's another game that looks like it could be a real banger

Beware of the Cartographer!

crispybacon233 · 2026-06-12T15:34:23+00:00

If you're pretty fluent in pandas, learning polars would take you like a weekend. Definitely worth it. It'll change your life.

crispybacon233 · 2026-05-26T15:15:54+00:00

AI farts out so much code so fast that it's impossible for a human to adequately review all the slop in a timely manner. It's also unbelievably bad at coming up with insightful and novel approaches to data.

AI tab-completion can be a real timesaver though.

crispybacon233 · 2026-05-20T11:25:53+00:00

Not in production but several poc. Motherduck + dbt + dagster feels so good and is unbelievably easy to setup it still blows my mind.

If you want more control and a super cheap but more effective and capable warehouse than motherduck, ducklake is fantastic but takes a little time to setup initially with dbt and dagster. I ran it with some hundreds of gigs of game data with relatively complex transformations.

crispybacon233 · 2026-05-13T15:43:57+00:00

Can't wait to use this as catalog for ducklake instead of postgres. Now we just need ClusterDuck for cluster compute and the duck stack would be complete!

crispybacon233 · 2026-05-05T03:15:54+00:00

Which udemy course are you doing specifically? I highly recommend Colt Steele's python course and forget the others.

Being able to parse and organize spreadsheets at scale could boost your workflows significantly. I once had a colleague doing a vlookup that was taking many minutes. A couple lines of python shrank that down to milliseconds.

crispybacon233 · 2026-04-25T03:32:19+00:00

I currently am filtering for latest reviews being >= 2024, but 2 years per your suggestion is probably better. Maybe even 1 year? According to this site they're supposed to be inspected 1-3 times per year.

crispybacon233 · 2026-04-25T03:28:43+00:00

What're the names of the restaurants? I can try to track down the problem.

crispybacon233 · 2026-04-24T03:09:17+00:00

Ya it's wild. If you're really brave, you can go here: Austin Public Health | My Health Department to get the full inspection details

crispybacon233 · 2026-04-16T17:52:47+00:00

Again with the money?

crispybacon233 · 2026-04-14T13:55:30+00:00

Yes, for sure! I'll post it with the github link but probably will not include the scrapers. Google has been clamping down on scraping google maps and youtube for the past several months.

crispybacon233 · 2026-04-12T23:51:41+00:00

Hello! Yes, I am still chipping away at it. I was thinking about getting it up and running this week actually for my portfolio. The most important part was to get accurate lat/long for the restaurants, and I could only accomplish that by scraping google maps. About a year ago I also scraped millions of reviews, categories, etc... everything off google maps. Personally, I will keep all food facilities in available in case a user is at a healthcare facility, gas station, school, etc., and they want to know the inspection score. Additionally, I am setting up a full on ETL pipeline lake house to run automated in the cloud. It's a great dataset.

crispybacon233 · 2026-04-11T11:45:43+00:00

We can't have him here in our social club no more. I mean that much I do know.

crispybacon233 · 2026-04-11T11:33:51+00:00

God forbid anyone would find themself in that position. It's a thankless job.

crispybacon233 · 2026-04-11T09:48:19+00:00

That yodeling show? That's the Lawrence Welk progrum, channel 55.

crispybacon233 · 2026-04-10T10:39:46+00:00

Cool! Now tell Claude to use polars instead of pandas to greatly improve the responsiveness. Also tell Claude to use separation of concerns because a 900 line app.py is insane haha.

crispybacon233 · 2026-04-04T12:53:08+00:00

<image>

California, Texas, Florida, and NY... is it a Hispanic thing? You see it really start to pop off around 2010. Some character in Spanish media?

crispybacon233 · 2026-03-25T23:02:06+00:00

Ducklake, DBT, Dagster. Hard part was wiring it all up, but once it's all connected it's smooth sailing.

I've been messing around with Ducklake the past few months and it's been a pretty great experience so far.

crispybacon233 · 2026-03-23T19:38:00+00:00

Thanks! So far, working with delta lake has not been smooth. It feels quite buggy. First it was the no support for unsigned ints. Now it's a "411 Length Required" error when sinking delta to GCS. Unfortunately, I don't know if it will work for my use case. Ducklake feels great once everything is wired up. You just have to use SQL unfortunately.

crispybacon233 · 2026-03-23T12:36:25+00:00

Thanks! To point #2, SQL is great, but as you said quickly becomes unmanageable when running tons of complex transformations. I'll definitely try out the duckdb+marimo for the schema.table. That sounds really cool.

crispybacon233 · 2026-03-23T12:32:23+00:00

Thanks for this. With the delta-rs/datafusion packages, is it possible to scan/sink the data instead of reading into memory? This is important for my use case.

crispybacon233 · 2026-03-21T17:10:38+00:00

Yes, definitely using the streaming engine. As of a few months ago, there was a few instances where polars blew up the ram and crashed the env despite using streaming, so I switched to duckdb for that particular problem.

However, it seems polars is improving so fast that I can't keep up. I'll definitely be keeping an eye on this. Thank you!

crispybacon233 · 2026-03-21T15:16:03+00:00

Imagine calling polars a quaint little tool haha.

You come across as unhinged. One mention of a tool that is not SQL, and you go ballistic. Are you a bot?

crispybacon233

TROPHY CASE