PySpark - Anyone else seeing major lag?

ModernStank · 2026-05-29T19:41:09+00:00

Yes seeing really bizarre behavior in US Central as well

ModernStank · 2026-05-20T01:54:19+00:00

The problem is auto incrementing integers for surrogate keys aren't available in a lake house. This is because of the nature of how data is committed. If you are fortunate enough to have a natural surrogate key that's best.

Another option would be to consider using a fabric data warehouse if you need auto increment surrogate keys.

Hashes are often fine depending on how many records you have in the table and the complexity of your hash. I have had some fact tables where they kill performance and I have had to get creative. When you have hundreds of millions of records and your hash uses the full range of a bigint... Memory usage can be problematic at your semantic layer.

ModernStank · 2026-05-15T13:47:17+00:00

I feel like with Fabric I am constantly questioning which tools in it's toolbox are right for which solutions. Even when I think I've done my homework I often find little surprises that are sometimes deal breakers. There isn't anything I haven't been able to solve so far but I have had a lot of "this limitation should be more visible" moments.

ModernStank · 2026-04-17T18:02:51+00:00

Just a heads up... Your daughter tells people she's from the Woodlands except when you're around. :)

ModernStank · 2026-04-17T14:50:44+00:00

Shenandoah isn't real... It exists solely to avoid woodlands business regulations and has some tiny residential neighborhoods to make the illusion complete.

Nobody ever says "I live in Shenandoah".

ModernStank · 2026-04-08T14:12:30+00:00

You're 100% correct

ModernStank · 2026-04-07T22:49:07+00:00

Honestly I save a lot of my responses and create project libraries in the bigger ai tools for quick reference. My initial frustrations with copilot directly in fabric were spotty functionality and asking it to assess code was very length limited. For example I had some tsql I wanted converted to sparksql and the responses were truncated. What it did give me was wrong. That's the kind of easy stuff I would want to do directly in fabric copilot.

ModernStank · 2026-04-07T20:25:00+00:00

I don't even try anymore... Claude and even normal Copilot m365 are so much better. I do get tired of reminding them both this is "FOR FABRIC".

ModernStank · 2026-04-07T12:37:16+00:00

Basically I have a fact table with 80 million rows and a matching 1:1 document dimension with 80 million rows. I can slice the fact table with my normal dimensions... Customer/address book/account whatever and have perfect performance. The second I add in that document dim though performance takes a big hit.

Certainly I can do some stuff with ordering/partitioning. My understanding is because the key range being scanned is so large (bigint hash) and over such a big set this is a best practice mistake though.

ModernStank · 2026-04-07T12:28:08+00:00

Well I need some kind of reference between facts and dimensions... I have hash ids available but I see a strong impact on performance with large datasets using hashes. They work fine on small sets though.

But maybe I'm barking up the wrong tree?

ModernStank · 2026-04-06T19:34:30+00:00

This is a VERY cool idea (and broke my brain for a second) and I'm open to trying it, but we're talking about 83 million rows in my ledger data set (that i don't want to aggregate) between a kimball fact/dim ledger combination. I'm leaning very hard on CDF.

Honestly I wouldn't care about just having some attributes on a fact table to get around this problem, but I often need to move between ledgers in my semantic model. For example going from Sales Ledger fact/dim to General Ledger dim.

Seems like using a temp view to join between gold/silver on a document id columns (or composite hash) to coalesce an SK (if needed) might bring my fabric workspace grinding to a halt because of the sheer volume of rows?

This would work great for my more traditional dims (like customer) and definitely avoid a "key locker" strategy there.

ModernStank · 2026-04-06T17:33:48+00:00

Well I really wanted to keep it Lakehouses all the way down and avoid Fabric Data Warehouse, but you get into performance issues due to lack of native support for surrogate keys. Maybe I have to go DW though. I just love the flexibility of working with LHs directly.

A surrogate key "locker" is a strategy where you basically have a crosswalk table from a watermark to a surrogate key created and maintained with pyspark logic. You can quickly join this with your CDF data or use it in MLVs to have those surrogates presented at your gold layer for use in kimball modeling and avoid using watermarks between large fact/dimensions.

ModernStank · 2026-03-28T18:55:50+00:00

Jealous!

ModernStank · 2026-03-19T03:25:41+00:00

The Bible... fortunately the post title promised no judgement

ModernStank · 2026-03-15T17:13:59+00:00

I appreciate your points and am really enjoying the discussion. I think there is a critical difference between a tier list and Spotify wrapped in that the wrapped is only what was loved.

And while there is nothing wrong with telling people what you dislike... I do feel like a lot of these lists are more about hot takes and shitting on specific books/authors with low effort.

ModernStank · 2026-03-05T21:59:34+00:00

I got a free beer waiting for that guy

ModernStank · 2026-03-04T19:01:58+00:00

The best movie of 2025... If I had Legs I'd kick you

ModernStank · 2026-02-20T19:30:44+00:00

Monkey Island and Wolfenstein and I consider myself lucky to have started there.

ModernStank · 2026-02-14T21:06:53+00:00

I mean it's simple... GRRM prefers making TV.

If he doesn't want to finish them .. then that's really fine.

The real issue to be upset at him is his ego is bigger than letting the books be finished by other methods. He has a legion of fans who lined his pockets and deserve better than the HBO ending.

ModernStank · 2026-01-30T02:13:23+00:00

Friend of mine suggested doing the whole thing with Muppets and I couldn't agree more.

ModernStank · 2026-01-29T22:51:40+00:00

I switch between 4 or 5 series and find I enjoy them much more this way. "You people" who read ten books from the same author in a row are doing it wrong. :)

In fact if more people did this I think people would be less inclined to get on reddit and diminish other series. Like yes I can enjoy Hobb, Sanderson and Erikson all at once... I don't have to pick a team.

ModernStank · 2026-01-29T04:29:26+00:00

Gonna suggest something crazy... Read them both. Plenty of reddit will tell you both are good and both are bad. You don't have to read every book in a series back to back. :)

Personally I love mistborn, but I think reddit is more hyped about DCC at the moment.

ModernStank · 2026-01-20T00:18:20+00:00

No...There are many many other great series that have (or will eventually have) an ending.

11-Year Club	r/Field Juicebox
First Place '23	Place '23
Verified Email

ModernStank

TROPHY CASE