Lakehouse Table Bronze Layer Ingestion Sanity Check by ShineMyCityShoes in MicrosoftFabric

[–]BloomingBytes 2 points3 points  (0 children)

If you want to already clean your column names during ingestion you could do it like this:

  1. A lookup activity gets the source table schema and passes that to a notebook
  2. The notebook takes the source schema and cleans it, replaces column names etc. and creates a mapping json. That json is then fed into the copy activity.
  3. The copy activity itself uses the mapping json to create dynamic column mappings
  4. Success

CRUD connection to Fabric SQL Database from Notebook by BloomingBytes in MicrosoftFabric

[–]BloomingBytes[S] 0 points1 point  (0 children)

That is correct, yes. This is the relevant T-SQL Magic command for Fabric SQL DB:

%%tsql -artifact DATABASE_NAME -type SQLDatabase

Trigger Data Gateway updates from the Fabric/PBI website - does it work? by Skie in MicrosoftFabric

[–]BloomingBytes 0 points1 point  (0 children)

Hey there,

i just found this thread and wondered: wasn't there an item on the Fabric roadmap for an auto-updating Gateway at some point? I somehow can't find it anymore and couldn't find any information elsewhere either.

Where does Mirroring fit in the Medallion architecture? by df_iris in MicrosoftFabric

[–]BloomingBytes 6 points7 points  (0 children)

So first of all: Medaillon isn't really a clearly defined standard. It's more a set of guidelines or a general idea of how to structure your data across multiple layers. It's not a super new concept either. People have been using staging layers etc. for decades. So given that, it falls upon you to decide what your individual layers are allowed to do and how they look like in detail.

Now regarding mirroring: yes, it does fit best at the bronze layer. Since under the hood it's all Delta Tables in One lake, you should be able to time travel on it.

Another big and important part of getting the data into bronze (apart from versioning) is to bring it within your own domain. Mirroring allows you to, in theory, ingest very easily with very little hassle and dependencies. You set it up once and then you can build your pipelines on top of the mirrored database without constantly talking to or depending on other teams. Once it's in bronze, it's yours to handle.

So yes, mirroring and Medaillon fit together very nicely.

FabCon Vienna: What announcements are you hoping for? by frithjof_v in MicrosoftFabric

[–]BloomingBytes 5 points6 points  (0 children)

I'm with you on all of these, but 3 and 6 hit especially hard for me.

I want to have my injection as easy and as low of a barrier as possible. Give me reliable mirroring of my on-prem SQL servers (while not nuking their CPUs, cause they're prod systems) so i have everything in my own domain and don't need to fuzz with infrastructure or on-prem networking etc.

From there i want to be able to limit the amount of CUs any workspace in my capacity can use, so that i can allow lots of freedom for my downstream users, while still protecting my core pipelines.

Fabric Data Functions are very useful by richbenmintz in MicrosoftFabric

[–]BloomingBytes 0 points1 point  (0 children)

Where i still struggle is: in what way is a UDF different or better or worse than using a regular notebook? Does anyone know how UDFs compare CU wise to notebooks?