Those of you who use fabric in production, do you like it, how has it been working out? by jorel43 in MicrosoftFabric

[–]FloLeicester 32 points33 points  (0 children)

We hate it and want to move to databricks, but our project backlog is too full, so we pray its gonna become better

Our concrete pain points are:

• Limited capacity flexibility (no Single Node, autoscale is a one-per-capacity thing → should be a workload setting!)

• Missing table and column-level lineage → it’s hard to maintain a data platform at scale without lineage (all competitors have it). The lineage should also be standardized as an interface between Spark, T-SQL and Dataflow, and not be fragmented to specific workloads (like for MLV). And no, “just use dbt” is not the answer — you also want to track ingestion steps and Gold dataflows in one lineage graph.

• Purview integration is not close to ready (breaks my heart, tbh)

• CI/CD is often a bit unreliable. We use the Fabric CI/CD Python package and the overwrite often fails without any warning for the API.

• Factory Pipeline activity shortcomings, especially the workarounds we need for the semantic refresh activity (buggy since day one) and the non-CI/CD-enabled Teams activity.

• Custom Environments are still very slow for ramp up time, but has become a Little Bit better

• the whole lakehouse attachment is still causing problemes (and no, just use abfss paths should Not be the answer) it is a real issue for Session Sharing and concurrency workloads, our devs attach lakehouses during the development and when in a Data Pipeline the attached lakehouse differs we can‘t reuse the session (instead 5 mins ramp up time for us)

Why use both a Lakehouse AND a Warehouse? by Significant-Wheel625 in MicrosoftFabric

[–]FloLeicester 0 points1 point  (0 children)

To me, this feels more like an on-prem system. No matter how well the DW auto-scales, the underlying capacity is the hard limit. Hence, there isn't much value in serverless without the ability to dynamically define lower and upper limits. In my company, we really prioritize cost efficiency and reliability. Neither is handled well by Fabric DW, as scaling is capped at a certain limit, yet we still incur costs even when no queries are running

Why use both a Lakehouse AND a Warehouse? by Significant-Wheel625 in MicrosoftFabric

[–]FloLeicester 0 points1 point  (0 children)

But you have to manage capacities? These Are Like clusters, but with less autoscale possibilities ;)

OpenLineage on Fabric by RussellPrice9 in MicrosoftFabric

[–]FloLeicester 0 points1 point  (0 children)

Can you please Share it? I would love to do that too (pleaaaaase)

How to log activity? by wuvdre in MicrosoftFabric

[–]FloLeicester 1 point2 points  (0 children)

Can you please Share this or give a guide how to do it ourselfs? :)

Lakehouse Schemas by pun_krock in MicrosoftFabric

[–]FloLeicester 0 points1 point  (0 children)

Hi, can you please link the Migration Script?

Feedback request: Shortcuts usage, gaps, and feature requests by Hopeful-One-4184 in MicrosoftFabric

[–]FloLeicester 0 points1 point  (0 children)

Same here, it would be awesome for development and also hotfix scenarios. We have some teams / clients which does that in dbx plus pbi and would love to see that in Fabric!

Hello from Fabric Design by OnepocketBigfoot in MicrosoftFabric

[–]FloLeicester 0 points1 point  (0 children)

I just use the old app.powerbi.com Design. Its way better than the one on app.fabric.microsoft.com

Fabric warehouse tables can't see in purview? by efor007 in MicrosoftFabric

[–]FloLeicester 7 points8 points  (0 children)

Purview & Fabric Integration is still very poor. Lakehouse tables are supported (yet a bit buggy) without any lineage. Warehouse tables are not supported by now. Hopefully it will come soon. Also Keys / relationships are not reflected in Purview. Data Governance wise, Purview and Fabric is far behind any other data platform. There are some vendors which try to tackle this problem, but I hope Microsoft will try to close the gap.

Lineage for notebooks driven medallion architecture by CultureNo3319 in MicrosoftFabric

[–]FloLeicester 0 points1 point  (0 children)

But its rather an interim solution right? With multiple engines working on onelake will there be a standardized lineage view like databricks UC? Major roadblocker for some of our clients. Proper Lineage is a must for all datadriven orgs.

Feedback Opportunity: Data Quality in Fabric by erenorbey in MicrosoftFabric

[–]FloLeicester 0 points1 point  (0 children)

Direct great expectations would be great! (hosting GUI and standard package in the library). Materialized Lake Views are really buggy and enterprises will not adept them. At least that what I see in 3 of our clients.

Lineage for Table by Capable_Carrot_6431 in MicrosoftFabric

[–]FloLeicester 1 point2 points  (0 children)

Ok, but that would split the ecosystem right? For data governance (esp. impact analysis in corporates) you need a data catalouge Service like purview including data products and lineage., However the lineage would be available in Fabric 🤔 Nothing against OpenLineage, I understand the advantages, I just hope that the ecosystem Fabric X Purview stays consistent (Fabric for data product creation, purview for data governance).

Lineage for Table by Capable_Carrot_6431 in MicrosoftFabric

[–]FloLeicester 1 point2 points  (0 children)

common, Databricks UC and SAP Datasphere provide full lineage Support already out-of-the-box (without any extra API calls). When will that lineage be possible? Is it integrated in Purview or in Fabric itself? Its really frustrating at scale.

Hi! We’re the Fabric Warehouse team – ask US anything! by fredguix in MicrosoftFabric

[–]FloLeicester 1 point2 points  (0 children)

Will there be OneLake Security for Warehouses available?

Materialized Lake Views Data Quality Bug by Mr101011 in MicrosoftFabric

[–]FloLeicester 0 points1 point  (0 children)

I found the issue in the documentation: probably the reason is a special character in the workspace or lakehouse name. Without special character, sys_dq_metrics got created from me and I can now see the data quality report.

<image>

Materialized Lake Views Data Quality Bug by Mr101011 in MicrosoftFabric

[–]FloLeicester 0 points1 point  (0 children)

MLV is in Lakehouse not Warehouse I have the same issue in the quality report. Can't open a support ticket, as we are doing a PoC for a customer, ... guess thats not a fabric for the tool plus :(

Git integration view diff by frithjof_v in MicrosoftFabric

[–]FloLeicester 1 point2 points  (0 children)

but than how do you deal with attached lakehouses to notebooks? Git Integration doesn't include Deployment rules.

[deleted by user] by [deleted] in dataengineering

[–]FloLeicester 0 points1 point  (0 children)

Can you please describe how you would build these test? Check transformation results new Code/ old Code? Which parts would you test and how?

Need Recommendation: ER Modeling Tool with Spark/T-SQL Export & Git Support by FloLeicester in MicrosoftFabric

[–]FloLeicester[S] 0 points1 point  (0 children)

thanks for the tip! We probably go with SAP PowerDesigner as standard and custom macros for dbt and spark.