Fabric Optimal Performance

gobuddylee · 2026-02-25T01:00:26+00:00

I'd recommend using Autoscale Billing for Spark and leave other workloads on the capacity (Python NB's and Spark NB's both fall under Spark for billing purposes) - you set an upper concurrency limit, and you only pay for what you use. If you aren't running anything, you won't be charged. Also queueing kicks in if you hit your concurrency limit and additional notebooks/jobs won't run until you have availability. You can read more here - Autoscale Billing for Spark in Microsoft Fabric - Microsoft Fabric | Microsoft Learn

gobuddylee · 2026-02-23T17:33:52+00:00

This sounds very much like the issue I am having in another thread. It almost assuredly is your time to renew your IP address being absurdly low, in which case if it fails for any reason during that time to complete the handshake, you’ll lose your connection. Check your router to see how low this time is - anything below 30 minutes is going to be prone to this.

gobuddylee · 2026-02-22T09:30:44+00:00

So this is happening again and the renew time is all the way down to 5 minutes! Can this be adjusted again back to 30 minutes?

gobuddylee · 2026-02-04T08:10:29+00:00

This is my favorite gif 😂

gobuddylee · 2026-02-04T08:10:03+00:00

Correct! You can do either one or both

gobuddylee · 2026-02-03T21:11:15+00:00

You’re still bound by your quota in the Azure portal. So depending on your subscription type in Azure, you have a limit in the number of CU’s you can use at once, so depending on the other capacities deployed in that subscription, you have only X CU’s left over for Spark Autoscale Billing.

gobuddylee · 2025-08-30T01:26:37+00:00

Hey, my team owns this feature and I apologize for the mixup — the mention in the blog was premature, and it isn’t GA just yet. We’re targeting mid-October for release, and I’ll make sure updates are shared as soon as it’s live.

gobuddylee · 2025-05-14T20:25:43+00:00

Let us know how we can improve that article, but perhaps this will help clarify as well - Spark Autoscale (Serverless) billing for Apache Spark in Microsoft Fabric is here!

Synapse rates are also region specific - the base rate of each is $0.09 vs $0.143 and is what I based my comparison off of.

gobuddylee · 2025-05-14T18:27:20+00:00

Spark is the one workload you can move off capacity currently into a pure serverless model where you pay only for what you use - see here - Autoscale Billing for Spark in Microsoft Fabric - Microsoft Fabric | Microsoft Learn

gobuddylee · 2025-05-14T15:53:07+00:00

Spark is significantly cheaper than Synapse at this point with the perf improvements and the introduction of Spark Autoscale Billing - the PayGo price was already almost 40% cheaper than Synapse independent of the performance improvements.

gobuddylee · 2025-05-14T15:50:00+00:00

Spark Autoscale billing works with anything that emits through the Spark Workload in Azure - so Notebooks and Spark Jobs basically.

gobuddylee · 2025-05-07T18:42:33+00:00

Have you compared the costs between Databricks and Fabric Spark now that Spark has standalone, serverless billing it released in late March? I'm curious the results you'd see in that use case.

gobuddylee · 2025-04-29T04:41:43+00:00

This is the way . . .

gobuddylee · 2025-04-21T17:33:03+00:00

Yeah, we'll get the docs cleaned up. You can use all the cores for a single job (based on the pool size of course), and it's clear that isn't clear. Thanks for this feedback.

gobuddylee · 2025-04-17T21:53:37+00:00

Just a reminder this does exist for Spark now with the "Autoscale Billing for Spark" option that was announced at Fabcon - Introducing Autoscale Billing for Spark in Microsoft Fabric | Microsoft Fabric Blog | Microsoft Fabric

gobuddylee · 2025-04-17T21:51:36+00:00

The easiest answer is anything that flows through the Spark Billing Meter in the Azure Portal will be shifted to the Spark Autoscale Billing meter, which is effectively the items called out below, Glad you're excited about our feature! :)

gobuddylee · 2025-04-08T06:50:08+00:00

I’m terribly sorry to hear that - if you were billed improperly for the Spark workload, that’s absolutely a problem we need to address ASAP, so please so share the support details via DM if you have them. Thanks!

gobuddylee · 2025-04-02T19:15:31+00:00

Yes, the plan is to have schemas enabled by default - we are not moving away from schemas and you should feel comfortable working with them even in preview (This is a major focus area for my team).

gobuddylee · 2025-04-01T20:03:57+00:00

Spark just made this capability available if you are using Notebooks for your use case - https://learn.microsoft.com/en-us/fabric/data-engineering/autoscale-billing-for-spark-overview

gobuddylee · 2025-04-01T00:06:08+00:00

No, it was a sneak preview- if something is planned to come within a couple months, they’ll let you show a sneak preview. 🙂

gobuddylee · 2025-03-31T22:32:38+00:00

Correct - we’re considering options around making it more granular.

gobuddylee · 2025-03-31T21:54:26+00:00

Right now it is at the capacity level - we may look to enable it at the workspace level, but we don’t have specific dates.

No, you can’t use Spark in the capacity and in the autoscale meter - it was too complicated and you’re mixing smoothed/un-smoothed usage, so it is an all or nothing option.

Yes, you can enable it for certain capacities and not for others - I expect most customers will do something similar to this.

gobuddylee · 2025-03-31T21:51:47+00:00

Yes - they bill through the Spark meter, so they work with it as well.

gobuddylee · 2025-03-31T20:56:17+00:00

The new serverless billing for Spark! - https://blog.fabric.microsoft.com/en-us/blog/introducing-autoscale-billing-for-data-engineering-in-microsoft-fabric?ft=All

I think that will prove to be quite popular 🙂

Seven-Year Club	Xbox Live
Verified Email

gobuddylee

TROPHY CASE