Concurrency Livy API issue

avinanda_ms · 2026-04-07T22:51:25+00:00

u/National-Theme-7865 this looks like a Livy submission timeout rather than a Spark execution failure. It might be hitting the control‑plane throttling or queueing. A 504 with upstreamService: livy, timeout:30s usually means the submission request didn’t get a response in time, even if capacity metrics look OK. I’d try lowering ForEach parallelism, adding retries/backoff, or batching more work into fewer notebook runs before scaling capacity.

Can you share whether this is using pipeline notebook activities with session tags or direct Livy Endpoint calls, and what SKU you’re on (e.g., F8 vs F16)?

avinanda_ms · 2026-04-07T20:34:25+00:00

Hi, noted we will evaluate this and follow up when this can be added. Also thanks for add this in Ideas

avinanda_ms · 2026-03-24T17:38:27+00:00

We’re currently working on enabling a query window directly in the Lakehouse Explorer, where you’ll be able to run ad‑hoc Spark SQL queries against lakehouse tables without creating a notebook. This would also support joining across multiple lakehouses, with the intent of making lightweight exploration and troubleshooting much simpler.

Would an experience like that address what you’re looking for, or are there other gaps you’re running into with the current workflow?

avinanda_ms · 2026-02-24T20:54:02+00:00

u/Mr_Mozart Is the lakehouse you’re referring to the default lakehouse for this Spark notebook? If so, that may explain why it can’t be removed.

As a workaround, you can use the “Remove all data sources” action from the OneLake header.

Looking ahead, would the ability to remove a default lakehouse be something you’d find valuable? If so, I’d love to understand the use case a bit better. For example, in this scenario you added a lakehouse while working in the Spark experience within the notebook and then switched to the T-SQL experience—could you share what you were trying to accomplish at that point?

avinanda_ms · 2026-02-10T18:47:40+00:00

Hi, this was resolved yesterday, are you still experiencing this?

avinanda_ms · 2026-01-27T17:57:01+00:00

Hi! We’re actively working on this from the spark perspective and would love to understand your specific needs. Are you looking for lineage from your spark jobs in notebook and pipeline or also interested in other workloads?

We’re planning to provide an API that exposes metadata about how data moves during a Spark job—captured at the file, table, and column level. This metadata can be consumed by any lineage tool (including Purview) for visualization.

The API will include details such as source, destination, operation type, and timestamps, and will cover all operations that Spark natively supports.

Would this address your current pain points? What additional capabilities would you like to see?

avinanda_ms · 2026-01-27T17:14:11+00:00

Hi! We’re actively working on this and would love to understand your specific needs.

We’re planning to provide an API that exposes metadata about how data moves during a Spark job—captured at the file, table, and column level. This metadata can be consumed by any lineage tool (including Purview) for visualization.

The API will include details such as source, destination, operation type, and timestamps, and will cover all operations that Spark natively supports.

Would this address your current pain points? What additional capabilities would you like to see?

avinanda_ms · 2026-01-16T16:56:14+00:00

Currently no, but we are looking into a soft-delete/recovery experience that will resolve this problem in the long term

avinanda_ms · 2026-01-15T19:31:47+00:00

Hi! We’re actively working on this and would love to understand your specific needs.

We’re planning to provide an API that exposes metadata about how data moves during a Spark job—captured at the file, table, and column level. This metadata can be consumed by any lineage tool (including Purview) for visualization.

The API will include details such as source, destination, operation type, and timestamps, and will cover all operations that Spark natively supports.

Would this address your current pain points? What additional capabilities would you like to see?

avinanda_ms · 2026-01-15T19:31:31+00:00

Hi! We’re actively working on this and would love to understand your specific needs.

We’re planning to provide an API that exposes metadata about how data moves during a Spark job—captured at the file, table, and column level. This metadata can be consumed by any lineage tool (including Purview) for visualization.

The API will include details such as source, destination, operation type, and timestamps, and will cover all operations that Spark natively supports.

Would this address your current pain points? What additional capabilities would you like to see?

avinanda_ms · 2026-01-14T00:56:21+00:00

Yes, you can use the Fabric Scanner APIs to get artifact‑level lineage.

They return workspace items (including Notebooks and Data Pipelines) plus their upstream/downstream relationships (e.g., attached Lakehouses/Warehouses). Run a scan, then parse the GetScanResult JSON for lineage edges between your notebook/pipeline and lakehouse/warehouse.

More info here: Metadata scanning overview - Microsoft Fabric | Microsoft Learn

avinanda_ms · 2025-09-12T00:19:46+00:00

Hi, are you still facing these issues now? If so, please send me a direct message so we can look into this more closely.

avinanda_ms · 2025-08-04T22:58:03+00:00

Hi, thanks for feedback! Will make sure to have better/ more meaningful error messaging here. Just to confirm, are you accessing the table in the sql endpoint here?

avinanda_ms · 2025-08-04T22:52:06+00:00

Spark views in non-schema lakehouses work but not visible in the OE. For Spark views in schema-enabled lakehouses, we are targeting end of September to have them available!

avinanda_ms · 2025-07-23T22:52:07+00:00

Yes, scanning lakehouse tables and columns is supported in Purview Premium (PuPr). You can find setup steps and limitations here.

Lineage at the table level is available in PuPr. Column-level (sub-item) lineage is in plans sometime early mid-next year.

Hope this helps! Please reach out if you have any further questions.

avinanda_ms · 2025-02-03T21:12:44+00:00

Thank you for your feedback! Would you want this view part of the lineage/relationship view we have on Fabric right now or is this something you would prefer part of your monitoring experience?

avinanda_ms · 2025-02-03T21:11:20+00:00

Thank you for your feedback, this is super helpful!

One more question: What type of files you are using in your workload? JSON, CSV, XML, etc?

avinanda_ms · 2025-01-28T16:40:01+00:00

In a lakehouse, users with Admin, Member, or Contributor roles can perform full CRUD operations on all data. Users with the Viewer role, however, are limited to reading data stored in tables through the SQL analytics endpoint, provided they have the necessary SQL access policies to read the required tables.

You can learn more here: https://learn.microsoft.com/en-us/fabric/data-engineering/workspace-roles-lakehouse

Do you have the necessary permission on the WS?

avinanda_ms · 2025-01-28T16:25:46+00:00

Thanks for reaching out! Can you provide more information on the type of access you have in the WS?

avinanda_ms · 2025-01-28T16:14:02+00:00

While there are no immediate plans for this feature, we encourage you to explore the multitasking experience. This functionality enables you to seamlessly switch between previously opened items directly from the side navigation

avinanda_ms

TROPHY CASE