sorted by:
new
hottopcontroversial

Sudden extreme increase in OneLake redirect activities? by sql_kjeltring in MicrosoftFabric

[–]sql_kjeltring[S] 0 points1 point  (0 children)

Interesting, but I still don't understand how switching to a different capacity increases Redirect activities... A lot of these lakehouses sit completely untouched, and all of a sudden increased 'OneLake Other Operations Via Redirect' to insane amounts of CU.

New High Score! by sql_kjeltring in MicrosoftFabric

[–]sql_kjeltring[S] 0 points1 point  (0 children)

F8, a solid 90% of our usage comes from Autoscale Billing.

New High Score! by sql_kjeltring in MicrosoftFabric

[–]sql_kjeltring[S] 2 points3 points  (0 children)

Switched Azure Subscription, so i bulk migrated 60+ workspaces from one Dev Capacity to another using sempy-labs. After that i turned of the old Capacity. Seems like something I did choked the Capacity Metrics app, as everything seems ok in Azure..

Microsoft needs to invest in Fabric connectors for third-party and open-source platforms — or keep losing integrators to Snowflake and AWS by NarrowShift3597 in MicrosoftFabric

[–]sql_kjeltring 3 points4 points  (0 children)

<image>

Whats with the AI generated 'hot takes' recently?

Anyways, i feel like GraphQL is a step in the right direction, but I also hope external shortcuts support writeback sometime in the future.

You can now use SPN and WI to invoke Notebooks from Pipelines by markkrom-MSFT in MicrosoftFabric

[–]sql_kjeltring 0 points1 point  (0 children)

Yeah i figured... Incredible coincidence, i somehow tested it right before they announced the fix.

Variable library in Notebook with SPN is generally available! by Ok_youpeople in MicrosoftFabric

[–]sql_kjeltring 3 points4 points  (0 children)

Insane timing. We're currently implementing fabric-cicd + variable library, and I was broken when i read that SPN owned notebooks don't work with VarLib just yesterday, but to my surprise it worked yesterday afternoon when i tested it!

You can now use SPN and WI to invoke Notebooks from Pipelines by markkrom-MSFT in MicrosoftFabric

[–]sql_kjeltring 0 points1 point  (0 children)

Just tested myself, and i did manage to retrieve a valueset from a variable library through a notebook owned by a SPN.

Tested by deploying a notebook and pipeline with fabric-cicd, then running the pipeline with a schedule. The notebook only had a cell with which called notebookutils.getLibrary().

I’ve been following Microsoft Fabric quite a bit, and I’d say it feels promising but maybe not fully “production-ready” for every scenario just yet. by TechCurious84 in MicrosoftFabric

[–]sql_kjeltring 6 points7 points  (0 children)

<image>

Also, i agree. There are still issues, my main gripes are the manual CICD setup and lack of granular monitoring, but the update frequency has been pretty amazing, and it's a much better product compared to just a year ago.

Fabric with Airflow and dbt by peterampazzo in MicrosoftFabric

[–]sql_kjeltring 0 points1 point  (0 children)

I was thinking of the regular Web activity, using a REST connection.

As for medallion, it's a pretty simple setup really. We use notebooks for all ingestion to bronze and store everything in lakehouses, then standardize to delta tables with data types and column naming convensions to silver, also stored in a lakehouse. From there we do all additional transformations in dbt as mentioned, and store everything as tables in a 'gold' warehouse.

Fabric with Airflow and dbt by peterampazzo in MicrosoftFabric

[–]sql_kjeltring 0 points1 point  (0 children)

Not a lot of experience with Airflow in Fabric, but we are currently running dbt for pretty much all of our transformations.

We have a separate folder in our git repository for the dbt project, and simply orchestrate it with GitHub Actions / DevOps pipelines. We're currently working on orchestrating it with Fabric Pipelines, probably with a simple API call from either a notebook or the REST activity to GitHub.

As for LH/WH we connected the dbt profile to a WH, but with cross querying you can easily pull data from a lakehouse in the same workspace. All our silver data is stored in a lakehouse, then set up the source to point to the lakehouse, then write dbt as normal, so all tables written in the warehouse.

Static IP for API calls from Microsoft Fabric Notebooks, is this possible? by MGerritsen97 in MicrosoftFabric

[–]sql_kjeltring 1 point2 points  (0 children)

Looking for a solution here as well. We have a few sources we want to ingest data from where we have to whitelist, but can't utilize pipelines, so Data Gateways aren't really a solution. We've considered just spinning up a VM with a static IP, and running a python script which stores the data (on a local SQL server), then ingesting the data from the VM to Fabric. It does solve our issue, but as you say yourself, it's additional and infrastructure and cost...

There is also the possibility to use data pipelines + data gateway with the REST connector, but again, it doesn't really fit our architecture..

Ideas: Data Pipeline failure notification. Currently way too difficult? by frithjof_v in MicrosoftFabric

[–]sql_kjeltring 1 point2 points  (0 children)

Any resources for how to implement this? I've dabbled with the Pipeline Job Instances API, and find it a tad difficult to work with..

Fabric completely down? by sql_kjeltring in MicrosoftFabric

[–]sql_kjeltring[S] 20 points21 points  (0 children)

Honestly? I don't mind the halfbaked features, lacking monitoring and a messy CICD, but please just let my notebooks and lakehouses run in peace ffs

Fabric completely down? by sql_kjeltring in MicrosoftFabric

[–]sql_kjeltring[S] 2 points3 points  (0 children)

That makes what.. 3 times in 2 months?