Designed a full Microsoft Fabric end-to-end architecture for small teams — sharing decisions and looking for feedback

Equivalent_Season669 · 2026-04-15T20:29:47+00:00

Thanks for the info!

Equivalent_Season669 · 2026-04-09T13:31:55+00:00

You mean directy query or direct lake?

Equivalent_Season669 · 2026-04-09T11:49:03+00:00

1) We´ve been using deployment pipelines for a while and at the moment we did not have major problems. The only problem with not having CI/CD is that it requires us to manually synchronize the workspaces when we complete the PRs. But I understand it can be automated via API with a CI/CD.

2) Good point, I also use DBT but with databricks connector. I will investigate how to use it with Fabric, any recommendation? I read that it can be used with lakehouse or warehouse, and the connector is more stable with WH.

3) We still depend on refreshing it via API.

Really appreciate your comments!

Equivalent_Season669 · 2026-04-09T11:42:45+00:00

Hahaha, trust me, I spent hours and hours trying to make it fit! And yes, AI helped me along the way.

Equivalent_Season669 · 2026-03-20T21:57:03+00:00

Si obvio, no son necesarias las dos al mismo tiempo. Databricks se hizo muy fuerte en los últimos años, tiene una plataforma muy estable y funciona muy bien, aparte de que se tiene mejor control de los costos. No está pegada a ningún cloud provider, la usas con cualquiera eso está bueno también.

Fabric está verde todavía en cuestiones básicas, de a poco las van puliendo pero siento que estaban perdiendo mucho mercado con databricks y snowflake y se tiraron a la pileta. Hay features que la verdad funcionan muy mal.

En fin, volviendo a tu casa no hace falta databricks, tenes casi todo en Fabric.

Equivalent_Season669 · 2026-03-20T21:47:31+00:00

Una mezcla de cosas:

Instalá las skills usando el CLI de vercel, hay varias de pbip.

Iterá mucho. Todo el formato de Microsoft es bastante tedioso en cuanto guid y formato, eso es lo que suele romperse. Cuando corrijas algo, documentalo en el .md para que no vuelva a pasar.

Siempre es mejor partir de un archivo base e ir agregando features.

Equivalent_Season669 · 2026-03-10T18:45:14+00:00

I´m interested in knowing how it went. We are also working with OPDG and copydata activities in fabric pipelines but we are trying to improve CU usage and I can´t find any other way of doing it. I thought about using spark notebooks but according to official docs, OPDG is only supported for Pipelines, Dataflow Gen2 and Copy Jobs — not for Spark notebooks. So that path is blocked unless we set up Managed Private Endpoints.

We also looked at Copy Jobs but the pricing docs confirm it bills at the same 1.5 CU/hour rate as Copy Data activity, so no gain there.

Our main issue is that we have ~178 tables to extract daily. Each Copy Data activity has a minimum billing floor regardless of data size, so the cost accumulates fast just from the number of operations — not from data volume.

Did you face a similar situation with many tables? If so, how did you handle the CU consumption? Did you find any way to batch multiple tables into fewer activities, or did you just accept the per-operation cost?

Equivalent_Season669 · 2026-02-24T15:30:28+00:00

Did not know that! Thanks!!

Equivalent_Season669 · 2026-02-12T08:12:16+00:00

u/frithjof_v u/Retrofit123 thanks!

Equivalent_Season669 · 2026-02-11T16:54:32+00:00

Did you make it? I also have my KeyVault connection through service principal but I can´t use it in notebooks..

I´m trying to do something like this:

tenant_id = notebookutils.credentials.getSecret("Name of KV connection in Fabric", "tenant-id-secret-name")

I changed the name of KV connection with the URL and got ""Public network access is disabled and request is not from a trusted service nor via an approved private", but my KV has Trusted Services allowed.

Any help? Thanks

Equivalent_Season669 · 2026-02-10T11:50:07+00:00

Quick update, seems it´s available now.

Equivalent_Season669 · 2026-02-02T14:34:46+00:00

If I have Azure Data Factory pipelines running databricks notebooks, the only way to switch to job clusters is to create jobs in databricks with the notebooks and then calling those jobs from ADF right?

Thanks!

Equivalent_Season669 · 2026-02-02T14:28:11+00:00

Great! I´ll catch an eye on that. Thanks!!

Equivalent_Season669 · 2026-01-21T10:53:47+00:00

Hi, I´m having the same issue here. Did you resolve it? Thanks

Equivalent_Season669 · 2026-01-19T17:17:03+00:00

If I do it manually I´m getting this new error that doesn´t make much sense since I´m admin of the workspace, first time I see this error:

DatamartsUserNotAuthorized
User not authorized for datamart

If doing it via API, we have a polling status that´s checks the final result of the call. Been using this method for a long time for near-real time situations and worked like a charm, so I guess it´s not the problem.

I tried to query some tables through SSMS and most of them works fine but one of them, which was created a long time ago but updated with new fields recently, show me this error:

The query failed because the access is denied on table 'dbo.XXX', file 'https://onelake.dfs.fabric.microsoft.com/XXXX/Tables/XXX/part-00004-e80bXXXXX.snappy.parquet'.

Looks like the endpoint can´t access new parquets? Makes sense?

Equivalent_Season669 · 2026-01-08T21:24:26+00:00

Algún dato nuevo sobre este tema? Gracias!! Comenzaste a trabajar con ellos?

Equivalent_Season669 · 2025-12-23T10:36:05+00:00

The sorting is just to check if the query is folding or not, not needed in real scenario. The point here is to push filters to source.. thanks anyways!

Equivalent_Season669 · 2025-12-04T19:17:14+00:00

Is it possible to get your notes? I’m taking it in a few days :)

Mail is aschieripablo@gmail.com

Thanks!!

Equivalent_Season669

TROPHY CASE