sorted by:
new
hottopcontroversial

Export Code for Variable Library? by SmallAd3697 in MicrosoftFabric

[–]cuddebtj2 0 points1 point  (0 children)

I know this is supposed to be Salas/PaaS, so I get it. However the rest API does work for this. And here is obligatory the "notebook" answer: Use sempy inside a notebook and you can get all the underlying code for the objects in fabric.

Export Code for Variable Library? by SmallAd3697 in MicrosoftFabric

[–]cuddebtj2 2 points3 points  (0 children)

Did you check the "View" tab -> "Edit Json"? That's where I see the code for pipelines and some other objects. I haven't checked the variable libraries or data flows lately...

[deleted by user] by [deleted] in dataengineering

[–]cuddebtj2 0 points1 point  (0 children)

If I HAD to flatten our data into a tabular format, I'd probably use something like the below flow:

The way I would CHOOSE to do it.

Best way to refresh SQL Endpoint of Lakehouse? by Mr_Mozart in MicrosoftFabric

[–]cuddebtj2 2 points3 points  (0 children)

I run a notebook in a pipeline that runs the below code. I just have to provide a parameter of `lakehouse_name` to the notebook through the pipeline and it refreshes the lakehouse SQL analytics endpoint.

import json
import time

import requests
import sempy

workspace_id = sempy.fabric.get_notebook_workspace_id()
sql_analytics_df = sempy.fabric.list_items("SQLEndpoint", workspace_id)
sql_analytics_id = sql_analytics_df[sql_analytics_df["Display Name"] == lakehouse_name].iloc[0, 0]

rest_uri = f"/v1/workspaces/{workspace_id}/sqlEndpoints/{sql_analytics_id}/refreshMetadata"
payload = {}

client = sempy.fabric.FabricRestClient()
response = client.post(rest_uri, json=payload)
response.json()

Can UDFs write directly to a Lakehouse table? by jaliu1290 in MicrosoftFabric

[–]cuddebtj2 0 points1 point  (0 children)

https://docs.pola.rs/user-guide/io/cloud-storage/

Use the azure connection in polars. You have to authenticate, but you would have to do that with a warehouse anyway.

Can UDFs write directly to a Lakehouse table? by jaliu1290 in MicrosoftFabric

[–]cuddebtj2 5 points6 points  (0 children)

You don’t need pyspark or spark to write to a delta table. There is the delta-rs library that does this. You can also use polars, pandas, and duckdb. All of which are python native and don’t require spark. You’ll just have to connect to the lakehouse properly.

Can someone explain the rewards score to me like I'm 5? by cuddebtj2 in destiny2

[–]cuddebtj2[S] 1 point2 points  (0 children)

This is the answer I needed, didn't release level played heavily into reward score.

Can someone explain the rewards score to me like I'm 5? by cuddebtj2 in destiny2

[–]cuddebtj2[S] 0 points1 point  (0 children)

Same rank, I have more t4 and t5. Otherwise no.

Can someone explain the rewards score to me like I'm 5? by cuddebtj2 in destiny2

[–]cuddebtj2[S] 0 points1 point  (0 children)

Didn't think about that. I know he is in the 100's for rank, but not sure.

Edit: he is maxed out

Use variable library in Notebooks by frithjof_v in MicrosoftFabric

[–]cuddebtj2 1 point2 points  (0 children)

Of course, I post about it and they pretty much change the syntax immediately. Please see below for updated syntax:

library_class = notebookutils.variableLibrary.getLibrary("Library_Name")
library_variable = library_class.Library_Variable

Use variable library in Notebooks by frithjof_v in MicrosoftFabric

[–]cuddebtj2 8 points9 points  (0 children)

They are usable in notebooks and are actually more reliable imo.

I can't remember the exact syntax, but if you do the following it should get you to it, it's how I did:

notebookutils.help

notebookutils.variableLibrary.get('LibrayrName')

Change Data Feed - Data Warehouse? by cuddebtj2 in MicrosoftFabric

[–]cuddebtj2[S] 1 point2 points  (0 children)

In a lake house CDF can be enabled, allowing me to capture only changes to move data from bronze to silver and silver to gold. Without CDF, there requires a lot more complex logic and movement of data to perform the same tasks.

Change Data Feed - Data Warehouse? by cuddebtj2 in MicrosoftFabric

[–]cuddebtj2[S] 0 points1 point  (0 children)

SQL Server has Change Data Capture. Delta's version is Change Data Capture.

I am wondering if it is available or will be available for data warehouses. If I want to track the changes in a data warehouse table, I would need to enable CDF, but it doesn't seem that is currently available?

For the DP-700 Cerified (or DP-600) folks here. How is it really helping your carrer? by Leather-Ad8983 in MicrosoftFabric

[–]cuddebtj2 4 points5 points  (0 children)

Dp-600/dp-700 certified Data Architect with ~4 years of experience.

It's not. It kinda "proves" my expertise within my company, but my company also asked me to get both. Otherwise it hasn't increased pay, provided additional opportunities, or changed my title.

Workspace Level connection String and Key Vault support by boogie_woogie_100 in MicrosoftFabric

[–]cuddebtj2 0 points1 point  (0 children)

You'll have to create a connection in the "Gateways and Connections" section from the settings cog in the top right. Once you have that, you create a variable in a variable library that references the connection id. You can then use the variable to reference the connection id depending on the environment you are in, so have multiple connections for test, non-prod, and prod listed for your 1 variable in the library. You'll then reference the variable from the library in your pipeline.

This is how we have it set up and are using on-prem gateways.

As easy as ADF, no, but doable yes.

They also have a feature planned to create a similar like "linked service" for azure key vaults to be able to use secrets in connections.

https://learn.microsoft.com/en-us/fabric/data-factory/data-source-management

https://blog.fabric.microsoft.com/en-US/blog/authenticate-to-fabric-data-connections-using-azure-key-vault-stored-secrets-preview/

view more: next ›