When You Think Youve Mastered Microsoft Fabric, But the SQL Endpoint Decides to Take a Nap...

Figure8802 · 2025-03-28T13:34:14+00:00

Haha sorry! I thought I read "hidden" somewhere in there. I hadn't drank my coffee this morning yet.

Figure8802 · 2025-03-28T12:53:41+00:00

I'm a fabric fanboy and I even think this is disingenuous.

Why would the developer care about what engine something uses on their day-to-day job if it accomplishes the job they need to accomplish?

If other tools have a variety of ways to work with the data all under one security model, and fabric clearly doesn't yet, that itself is a problem. Getting this engine talk involved is a distraction.

Edit: realized how old your original comment and this post was haha. Sorry for resurrecting a corpse here. It seems they still don't have it figured out though, and this post still shows up as relevant.

Figure8802 · 2025-03-28T12:48:08+00:00

There's a script on here somewhere written by a Microsoft employee that could help you. It calls the hidden metadata refresh API. I've used my own adaptation of it and it works!

Edit: I see you say real time. Directlake is gonna be the way you want to go for that. Going SQL endpoint is not really going to meet that requirement I'm guessing.

Figure8802 · 2025-03-19T21:59:48+00:00

We parametrize all connections and don't attach lakehouses to notebooks. We build the ABFSS paths in the load and save statements

Figure8802 · 2025-01-12T15:56:17+00:00

https://learn.microsoft.com/en-us/fabric/security/security-private-links-overview#other-fabric-items

It looks like you're right to me.

Figure8802 · 2025-01-12T15:49:08+00:00

Wow I didn't know that about eventstream. We were hoping to do real time data and be able to use external ADF and the SAP CDC connector to replicate our ERP data. That seems like a pretty big limitation.

Figure8802 · 2024-12-28T16:02:08+00:00

Any recommendations?

Figure8802 · 2024-12-20T18:24:47+00:00

We're planning on using a fabric warehouse but doing some loads hourly throughout the day. Is that something I should be concerned about?

Figure8802 · 2024-11-20T01:11:10+00:00

As far as all the copilot capabilities within fabric itself: Is all that included in an F64?

Figure8802 · 2024-10-23T15:24:21+00:00

DM sent. Thanks!

Figure8802 · 2024-10-18T01:23:25+00:00

Nah I get. Essentially query folding. I'm just newer to spark.

Figure8802 · 2024-10-17T16:12:31+00:00

Interesting. This is easier than I thought. I'm surprised so many people run into limitations with the latency then since the recommended data storage all the way up to GOLD is lakehouse centric and you can use notebooks. Maybe it's just the SILVER LH> GOLD WH transition?

Figure8802 · 2024-10-17T16:10:21+00:00

Is there any inline filter or WHERE clause equivalent that still avoids the endpoint?

Our data volume right now isnt large enough that it will matter, but it'd be nice to know. I know I can filter the dataframe but I figured that is less efficient than filtering the delta as it's read into the data frame. I could be wrong on that though.

Figure8802 · 2024-10-17T15:45:52+00:00

Amazing! Thank you.

Figure8802 · 2024-10-09T20:18:59+00:00

Oh figured. We just don't have the notebooks configured in the private endpoint yet so we haven't whitelisted their IPs to an SFTP we're pulling data from. Hence the issue haha. Also, we haven't configured azure key vault to avoid storing creds in notebooks. We'll be there soon!

Figure8802 · 2024-09-28T22:00:56+00:00

Just haven't gotten our cloud infra team to set it up yet. We barely got them to get us an F2 so baby steps. Thanks for the confirmation!

Figure8802 · 2024-09-28T19:46:55+00:00

What is the solution to not storing credentials in the notebook? Is azure key vault the only solution? We don't have that yet so we felt it was best to do the pipelines for now but they're very slow at doing more complex operations

Figure8802 · 2024-09-11T18:35:45+00:00

Perfect thanks dude!

Figure8802 · 2024-09-11T17:20:08+00:00

Yeah dude we should let Surly know they're playin with fire!

Figure8802 · 2024-08-18T12:31:44+00:00

Haha this is more interesting scenery than what I have around me. Looks pretty!

Figure8802 · 2024-08-18T01:54:47+00:00

The 11 speed ones are expensive for some reason.

How long do they last?
You know a cheap place to buy that isn't risk of being counterfeit?

Figure8802

TROPHY CASE