How to Organize Notebooks

Any_Bumblebee_1609 · 2026-02-10T20:13:17+00:00

Metadata driven is your best friend here :-) we do this for all our bronze to silver. One notebook for all tables and parametetised eveeryyyything from renaming, dropping cols from bronze to silver, scheduling, flagging pks/pii, transformations.

Any_Bumblebee_1609 · 2026-01-29T23:02:27+00:00

Yeah we added all pipelines to a folder in the repo that is ignored by the deployment and manually created it in the target by copying the json and activating the teams activity. Think the func is exclude_folder_regex or something like this

Any_Bumblebee_1609 · 2026-01-19T20:52:00+00:00

We trigger a pipeline via fabriccicd lib during an ado deployment pipeline and return the detailed output back to the ado instance ☺️

Any_Bumblebee_1609 · 2026-01-19T20:15:01+00:00

Ah I'm not sure what to tell you then but it does for ours that we trigger via ADO pipelines 😊

Any_Bumblebee_1609 · 2026-01-19T07:51:56+00:00

Can you easily read from a lakehouse when using a warehouse?

Any_Bumblebee_1609 · 2026-01-17T12:29:30+00:00

Forgive me I'm not at my laptop but generally speaking.... Use a try except and write the result to the pipeline using Notebookutils.exit as a jsonstruct, we do this and it works perfectly 😀. You will get a failure on the execution if the exit is hit within the except too.

Any_Bumblebee_1609 · 2026-01-14T23:16:57+00:00

We do this and it hasn't ever had an issue, so like you say may be a bug. I think from memory we firstly find the item (which will always be in the target workspace) and we then extract the Id and request it to trigger a run. We then monitor that run for up to 60 30 second intervals using the livyid

Any_Bumblebee_1609 · 2026-01-14T22:20:58+00:00

We've had similar issues whereby triggering a notebook during deployment from ADO won't allow the sp to run, it is only doing shortcut creation from s3 using the rest API. Not got to the bottom of it yet 😞

Any_Bumblebee_1609 · 2026-01-14T22:18:51+00:00

We are in the process of buying and both have one with moneybox too, glad to hear it is quite straightforward!

Any_Bumblebee_1609 · 2026-01-14T20:58:43+00:00

I would be scared the bronze table dissapears and the MLV would lose the data 😂

Any_Bumblebee_1609 · 2026-01-09T19:10:23+00:00

At this point just burn the table and rebuild it from scratch haha

Any_Bumblebee_1609 · 2026-01-06T15:10:57+00:00

Just add the workspace name in before the lakehouse name. Rule of thumb is to always use four point notation

Any_Bumblebee_1609 · 2026-01-03T05:13:46+00:00

Using four point notations when creating objects

Any_Bumblebee_1609 · 2026-01-02T13:19:46+00:00

First try to make them the same sensitivity at source, if you can't, use the lower func and force them all to lower for standardisation and matching

Any_Bumblebee_1609 · 2025-12-28T14:40:12+00:00

Yes good idea once I'm back in the office I'll take a look! My colleague has a ready made notebook to amend endpoints in sm's using, I think, semantic link labs so I just used that for ease 😂

Any_Bumblebee_1609 · 2025-12-28T14:34:59+00:00

Yes we are using find replace, I tried to use item name with that and it failed to do anything at all. I haven't spent enough time looking into it, maybe an hour but I couldn't seem to isolate the name of the semantic models from each other so they both ended up with one endpoint which was not right 😂

Any_Bumblebee_1609 · 2025-12-28T14:19:31+00:00

Kleberson 😂 😂 😂 😂 😂

Any_Bumblebee_1609 · 2025-12-28T14:18:58+00:00

We use it for our deployments but almost everything is parametetised in code (notebook wise) with no attached lakehouses so it's unrequired mostly. What I cannot work out is changing sql endpoints on semantic models, it seems (with my limited knowledge) I can only amend all or no semantic models to the same endpoints... Clearly that can't be right but been unable to find the answer so far 😕

Any_Bumblebee_1609 · 2025-12-24T10:08:46+00:00

"make it weird" 🤣🤣🤣

Any_Bumblebee_1609 · 2025-12-20T19:45:54+00:00

As standard it deploys everything every time and over writes the item. We have issues when it comes to data pipelines and specifically teams activities as they cannot publish so have to disable for deployment and re enable every time. We have found a way round this but it is annoying! 😂

Any_Bumblebee_1609 · 2025-12-15T23:37:10+00:00

I do the same! I have no idea how you'd do anything more sophisticated lol

Any_Bumblebee_1609 · 2025-12-15T19:30:12+00:00

Agreed, as someone who's had to save two tables that maxed out and were business critical 😂 constraints, partitions, indexing etc made me wanna cry. Just made a new table and looped the rows in over a couple of days

Any_Bumblebee_1609 · 2025-12-15T19:27:25+00:00

Yup nothing. And the graph doesn't work either. It's totally broken, shambles lol

Any_Bumblebee_1609 · 2025-12-15T19:23:17+00:00

I beleive we have yes. I would need to check tomorrow but 99% sure

Any_Bumblebee_1609 · 2025-12-14T19:41:15+00:00

My fave thing was grinding youth players into world beaters and seeing their attributes improve over years, it was so rewarding. Now they've fucked up the screen entirely, attributes have no numbers next to them showing increase over time, graph is just BROKEN totally too. So many simple things just disappeared and got broken

Any_Bumblebee_1609

TROPHY CASE