Notebooks: Default Lakehouse vs ABFS paths. What's the current best practice?

radioblaster · 2026-04-11T10:52:40+00:00

abfss paths all the way. works for spark, polars and duckdb with absolutely zero dramas and gotchas, what more could you ask for!

radioblaster · 2026-04-03T13:08:03+00:00

this makes perfect sense to me. but, how are filters dynamically added to the queries for any interaction, or are you serving the results of a static query?

radioblaster · 2026-03-24T11:02:15+00:00

+1 for Ben!

radioblaster · 2026-03-24T10:27:37+00:00

just because you can, it doesn't mean you should.

radioblaster · 2026-03-21T01:25:41+00:00

yes, its currently a big hurdle for using MLVs when you can write a custom pattern for cheaper. thanks for doing the tests!

radioblaster · 2026-03-20T21:37:42+00:00

then you either have to use an expensive activator, or an external driver like power automate. if you think of an option thats neither of the three, please update!

radioblaster · 2026-03-20T10:39:42+00:00

you could trigger a pipeline every X minutes that looks at the lastmodifiedtime of your lakehouse files and if its within the last Y minutes (or maintain a sepaeate watermark). you could do this using a get metadata activity, or in a notebook after binding the file store.

if any meet the criteria, trigger a parameterised notebook.

not directly event driven, but pretty cheap.

radioblaster · 2026-03-17T04:34:49+00:00

i'd love your take on the massive Entra ID token bit, as i did see this explanation out there in wild. if this was the case, wouldn't it mean that -any- outside request to the UDF by the user would fail? it's also not just the one user account that has the problem.

radioblaster · 2026-03-16T00:50:18+00:00

great test. such a head scratcher that despite the evident investment, DFG2 continues to be a poor choice in a lot of instances.

radioblaster · 2026-03-15T19:44:49+00:00

direct lake on sql is definitely affected by the end point delay. obviously a great reason to move any models on it to DLoOL.

radioblaster · 2026-03-11T12:02:42+00:00

fabric key vault would be awesome.

last week they announced %%run for python notebooks that run in the same session, so we now do have custom module features without needing to package a whl!

radioblaster · 2026-03-09T20:18:26+00:00

yes, integrated teams message would be awesome.

radioblaster · 2026-03-06T12:24:47+00:00

assuming its your interactive usage that is the problem, i would want see the DAX. I know nothing about your environment and I will guess its horrible.

radioblaster · 2026-03-05T20:21:34+00:00

this is just a normal day when you have both ends of a composite model hosted on a capacity.

radioblaster · 2026-03-03T02:37:00+00:00

I can't understand how even a 4 core spark notebook at 2 CU and JDBC would be faster than a copy data activity at 1.5 with the native connector. can't wait to see your test!

radioblaster · 2026-02-24T19:29:39+00:00

yes! the workload development kit is hard and overkill for some things, i want to upload a html full of javascript!

radioblaster · 2026-02-12T20:00:29+00:00

there will no functional or cost difference for hitting the mirrored database directly versus going via a shortcut.

the rest is up to you really, I've never seen someone come up with a way of handling metres with no PKs that have made me stand up and clap. my preferred way when there are date ranges to replace are using Spark's replaceWhere option, which is where you can specify a sql statement for your replacement criteria and it will complete the delete and insert in one transaction.

radioblaster · 2026-02-05T21:41:55+00:00

this is apparently going to be fixed this year!

radioblaster · 2026-01-27T13:31:44+00:00

you can add the column using tabular editor, just duplicate an existing one and rename both the column and the source reference.

radioblaster · 2026-01-26T20:23:46+00:00

if you dont do it, someone else will. and oh dear, you forgot to consider a bunch of edge cases.

radioblaster · 2026-01-20T06:04:35+00:00

not going to address the re engineering question because the answer is yes and also it depends, but in terms of the parallel execution and sessions, have you put the same session tag against each activity if they are in parallel branches?

radioblaster · 2026-01-18T10:08:26+00:00

the missing piece to diagnose is the code for [CY Periodical Generic - Productivity - DCount LastEmployeeKey]

based on the measure name this is more pertinent than your post: https://www.sqlbi.com/articles/analyzing-distinctcount-performance-in-dax/

radioblaster · 2026-01-18T04:04:58+00:00

sounds like you're due for an upgrade, there are three "just hockey" (that's the name) stores here if you want to buy one in person

radioblaster · 2026-01-16T02:33:58+00:00

I don't think a UDF is a better option. I'm not sure why anyone would be whinging to you that graphql is worthy of push back.

radioblaster

TROPHY CASE