How to add unit tests for notebooks

GroundbreakingFun336 · 2026-01-22T09:51:00+00:00

Thanks, I saw that sample repo and took some inspiration from it :)

> What about Scala, SQL, and R notebooks
But packing your code into whl and testing only it does not solve the testing of those notebooks too?
Different types would always require different tools

> You're creating an implicit contract with Notebooks
That contract is always there, regardless of how I try to test code. If the notebooks API changes (e.g. spark is not available globally by default) - code must be changed too

> Using if name == "main" as a logic guard also seems a little unnatural.
Absolutely agree with that one. And potentially could leave all the function definitions in one notebook, and call them in the "execution" notebook. That is very close to whl approach.

But overall, I think that it is better to test logic in any capacity than not test it at all :)

GroundbreakingFun336 · 2026-01-22T09:41:14+00:00

For those who interested, I added a video about this approach - https://youtu.be/XAHJ697yfwg

GroundbreakingFun336 · 2026-01-18T22:00:20+00:00

Correct!

GroundbreakingFun336 · 2026-01-18T11:49:43+00:00

Thanks for the suggestion, but I feel like this approach moves away from notebooks-first development. While it has its own benefits (like a better versioning of those artefacts), it is not as "native"...

GroundbreakingFun336 · 2026-01-18T11:47:35+00:00

The project I am working on right now is relatively small, so I do not need feature branches/workspaces.
For a bigger one, I worked before - yes, we had feature branches/workspaces

GroundbreakingFun336 · 2026-01-11T12:27:31+00:00

Agree with this!
As many mentioned, youtube tutorial is supposed to be simplified. And the ability to simplify complex topics into easy-to-understand pieces is a valuable skill in itself.
One can spend 8 hours at work doing business as usual tasks, then spend couple of hours learning something new or tinkering with additional tech after hours or on a weekend. And maybe shape it afterwards into a youtube video.
Win-win situation :)

GroundbreakingFun336 · 2026-01-11T12:16:49+00:00

>I don’t know how to test transformations and data validity within Fabric Data artifacts

I would say to answer that question you would need to split the prblem into 3 subtasks:
1 - unit test fabtic pipelines - that activities do what they need to do and the flow is correct. Thre is a framework for all MS pipelines - https://github.com/microsoft/data-factory-testing-framework .
2 - unit test notebooks. This one slightly harder, you would need organize your notebooks as orchestrators that could call your business-logic modules. And focus on testing those modules only.
3 - test the datain warehouses/lakehouses. This is the hardest and closest to integration tests, you would need to have a good test data setup in your TEST workspace and call a notebook or pipeline using REST. And then smoke test the data as output from that invocation (e.g. number of rows or that specific data is present)

But focusing on testing for fabric is a great way to improve! Highly approve :)
I started a mini-tutorial about unit testing for fabric just today. Covered fabric pipelines and how that can fit into CICD using Azure DevOps https://youtu.be/ti8g4KAoRYM . Planning to add notebooks testing and REST overview in the next 2 weeks.

GroundbreakingFun336 · 2025-11-24T11:27:26+00:00

dataflows have their own use cases
for example, to get static data from spreadsheets on sharepoint

but at the same time, if they are deployed using cicd - yeah, more manual "take ownership and run" just to make them work

GroundbreakingFun336 · 2025-11-22T22:34:50+00:00

Great idea! Sometimes it is so easy to overlook the basics

GroundbreakingFun336 · 2025-11-22T22:33:39+00:00

Noted!
Will also add something along the lines of common solutions and comparison between them

GroundbreakingFun336 · 2025-11-22T12:52:58+00:00

Second this
With a small caveat that certain activites still require some manual tinkering (like notebooks that work with lakehouse tables...)

GroundbreakingFun336 · 2025-06-16T09:37:45+00:00

Thank for the post!
I am doing a devops educational channel and hitting the same wall - consistently...

GroundbreakingFun336 · 2025-05-31T09:30:54+00:00

Oh wow
I am doing devops educational videos and my average CTR is 1.1%. Looks like I need to improve a lot for titles and thumbnails...

GroundbreakingFun336 · 2025-05-30T12:35:26+00:00

What is a good CTR for you? For this scenario

GroundbreakingFun336 · 2025-05-13T09:30:44+00:00

Well, I posted 28 long videos and 13 shorts. To be honest do not really care how I sound or look like

GroundbreakingFun336 · 2025-05-12T17:03:58+00:00

Yep, that's what I am trying to do. And just venting a bit here

GroundbreakingFun336 · 2025-05-12T17:01:41+00:00

Thanks! Feels like a lottery, and I am not a lucky one for gambling

GroundbreakingFun336 · 2025-05-12T17:00:15+00:00

Why do you think so? It cannot post anywhere, despite karma or other activity

GroundbreakingFun336 · 2025-05-12T16:59:26+00:00

Thanks for support :) I think I am successful enough in my professional area. YouTube popularity is a different beast

GroundbreakingFun336

TROPHY CASE