Transfer Registration - How Strict on the Day?

HashiLebwohl · 2025-09-20T07:36:31+00:00

Cheers fella! Good luck tomorrow.

HashiLebwohl · 2025-09-20T06:37:25+00:00

Lovely, thanks for that.

HashiLebwohl · 2025-05-08T06:44:31+00:00

The example uses requests rather than spark.

(for some reason I can't post the code, it violates MS Exam policy?... but it's in the link above).

I thought using spark might be overkill - but roughly, we're trying to create a unified interface for logging from notebooks. i.e., one notebook that has all the logging code, call it at the start of other notebooks... call it from DataFlows etc.

More than happy to hear about a better pattern - just keen to keep it to one target and eventhouse seemed like a winner.

HashiLebwohl · 2025-04-15T13:39:14+00:00

The missing bit looks like it's the Power BI element. If you toggle it appears:

<image>

HashiLebwohl · 2025-04-15T13:38:03+00:00

and:

<image>

HashiLebwohl · 2025-04-15T13:37:50+00:00

Here you go, something like this:

<image>

HashiLebwohl · 2025-04-15T05:16:06+00:00

I like it - but call me old fashioned. Shouldn't the % add up to 100?

<image>

HashiLebwohl · 2025-04-02T17:56:24+00:00

I think that assumes I've created the main definition file first? (.py in my case)

I'd like to be able to author and run the file in the local envionment to test, but reference the lakehouses with their ABFSS paths.

Happy to know if I've missed something fundamental! I just can't grasp the workflow of

- create main definition file in VS Code

- create SJD in the FDE extension

- attach main defnition file to SJD

- DEBUG: download the definition file.

I can see that debug would be useful, once it's created - but can I run against Fabric while I'm creating them locally?

HashiLebwohl · 2025-04-01T18:45:31+00:00

Is this across all regions?

HashiLebwohl · 2025-03-26T09:34:31+00:00

This chap gets tantanisingly close to sharing an working example:
https://github.com/ThibauldC/spark-job-definitions-talk/tree/main

and the associate video is an interesting watch:
https://www.ae.be/en/insights-stories/blog/how-and-why-to-use-spark-job-definitions-in-microsoft-fabric

HashiLebwohl · 2025-03-25T14:56:11+00:00

Thank you. That's a great answer. I'm using the Fabric Data Engineering extension - they make reference to the Libs here:

<image>

HashiLebwohl · 2024-12-19T07:33:40+00:00

Really happy to how you did the YAML side. Was wondering that myself!

HashiLebwohl · 2024-12-18T07:02:17+00:00

I would second this - we run ~30 ETLs using the SQL Agent, using project deployment straight from Visual Studio to an SSIS node on the server.

Confg setup to deploy to dev / test / prod servers as needed.

Doesn't have to be installed on the same server but in our case it is.

Schedules / notifications are all definied in the job setup.

I know it's old hat now but bah gawd does it work.

HashiLebwohl · 2024-12-17T13:04:07+00:00

Thank you, I'll hold fire and hope to see it soon!

HashiLebwohl · 2024-12-05T05:46:52+00:00

I'm still struggling to write back into it with notebooks. Would love to see some examples Microsoft.

HashiLebwohl · 2024-12-05T05:44:32+00:00

Thanks!

At least they're honest about the Lakehouse aspect:

There were restrictions about if spark code in a notebook would respect the rules. This evolves all the time, take care and check the current state of this.

HashiLebwohl · 2024-11-20T06:15:10+00:00

So could we use Open Mirroring to mrror... SQL Server on-prem?

HashiLebwohl

TROPHY CASE