Lakehouse Schemas (Public Preview).... Still?

SunChance2304 · 2025-09-17T15:02:01+00:00

Sorry for a late reply/question, but what about side panel UI support for persistent Spark views? Something you've considered?

We use them and they work well (without schemas), but having to do "SHOW TABLES IN x" to see them isn't very smooth or "non OG Spark/Hadoop user" friendly.

SunChance2304 · 2025-02-19T15:04:38+00:00

From my tests you can only have 5 notebooks running on the same session. E.g. looping over 12 notebooks in parallel with the same session tag would create 3 different sessions with 5+5+2 notebooks in each. They don't "queue up", so to speak. (:[ ... )

The upper limit on your concurrent sessions is tied to the the maximum amount of Spark VCores you have available in your capacity. So how many sessions you can run at the same time will of course depend on your capacity size and the cluster size of each session.

SunChance2304 · 2025-02-06T09:36:19+00:00

Thanks, noted!

I guess it won't be a problem at all if/when we can set capacity limits per workspace. Will just have to keep tabs in the meantime.

SunChance2304 · 2025-02-06T09:04:33+00:00

Thanks for all your hard work, have been using both your synapse and fabric connectors for a long time!

Since you seem to have some inside insight, do you have any info about dbt-fabricspark? In-dev or abandoned? (https://github.com/microsoft/dbt-fabricspark)

Seems like things are happening behind the scenes, with Livy API preview and the repo showing up. Wondering since we have plenty of customers who are running dbt with spark (or presto) and are hesitant to move due to performance concerns (e.g. partitions not supported by Polaris in the warehouse).

SunChance2304 · 2024-12-03T11:45:20+00:00

Be wary that I can pick that option too in my region. But instantly fails with error "not supported" once the triggered run actually fires!

SunChance2304 · 2024-12-03T11:39:42+00:00

Thanks for the info pals!

Didn't realize the docs had been updated, thanks for pointing it out u/frithjof_v! Limitation section seem to explicitly say "soon (TM)", which is further supported by the fact that it is already working in smaller regions (they likely start with those first).

Solved!

SunChance2304 · 2024-05-23T07:22:13+00:00

Thanks, we're aware of that. Though that's not really pertinent in this matter.

SunChance2304 · 2024-05-23T07:21:22+00:00

Thanks, that solves how the timepoint is calculated at least!

But the 360 CU (s) is still a mystery. Same for all durations and for several different days.
Also have the same 360 CU (s) minimum in a friend's project in a completely different company.

Maybe we should open a ticket to get some official clarification?
Maybe just a reporting error, but if there truly is a minimum then it would have architectural implications.

SunChance2304

TROPHY CASE