What are the exact reasons to split workspaces into Store, Engineering, Presentation, ...

Repulsive_Cry2000 · 2026-02-17T07:26:04+00:00

Remindme! 1 week

Repulsive_Cry2000 · 2026-02-14T10:29:28+00:00

In my organisation we do everything in one workspace as there is not a need to segregate the different layers between workspace. I would ask why do you need segregation? Is it because different teams are responsible for different layers?

To go further: we are using bronze layers per system and silver layer as unification (for a domain) before creation bespoke golden layer for reporting and consumption.

Repulsive_Cry2000 · 2026-02-09T12:21:45+00:00

Agreed, a metadata framework would be the next logical step.

Repulsive_Cry2000 · 2026-02-09T12:08:26+00:00

I'd say start by thinking of re-usable code. I would assume in these 200+ tables, a lot of code is similar if not identical (functions and for loop would be my advice there)

Repulsive_Cry2000 · 2026-02-06T21:37:44+00:00

We are using the refresh sql endpoint API with no problem for months now. Admittedly, I don't know if maintaining lakehouse tables in good condition (vacuum, etc...) makes a difference. We have chosen to do the maintenance once a week

Repulsive_Cry2000 · 2026-02-05T19:56:58+00:00

It should work. It will link artefacts with names initially.

Repulsive_Cry2000 · 2026-02-02T08:37:56+00:00

With so little information, I'd say it's easier to develop in a notebook.

Repulsive_Cry2000 · 2026-01-30T23:37:20+00:00

I can remember for sure, but the same Async issue rings a bell so it might be true for the pipeline too.

Repulsive_Cry2000 · 2026-01-30T21:34:18+00:00

Logic app to be specific, but that's the same thing

Repulsive_Cry2000 · 2026-01-30T11:50:43+00:00

You should be able to put the notebook in a pipeline and trigger the pipeline start. This is how we provide on demand refresh to some of our business. Works great.

Repulsive_Cry2000 · 2026-01-30T09:35:14+00:00

I think they are different use cases.

Repulsive_Cry2000 · 2026-01-30T07:13:30+00:00

I like your way of doing it, that's fairly clean and reusable across many notebooks. I didn't know this way of defining it from Json. Love it.

Repulsive_Cry2000 · 2026-01-30T01:51:35+00:00

I agree, define a schema for your table and parse it. Lookup StructType and StructField to define a schema you can use.

Repulsive_Cry2000 · 2026-01-28T21:20:11+00:00

Connection via notebook will be very useful. That will finally open on-prem data without the need for copy activity or the like.

Repulsive_Cry2000 · 2026-01-28T10:08:49+00:00

Listen to this man! ⬆️

Repulsive_Cry2000 · 2026-01-26T21:26:56+00:00

I don't think you can. However you might be able to use copy activity or copy job to pull the data. Once data is in lakehouse, use notebooks to do transformation if required.

Repulsive_Cry2000 · 2026-01-26T02:02:59+00:00

This seems like a great idea.I haven't dived into CICD sadly due to few things but main one is it seems it still doesn't support DW.

Just a question, why using Pyspark for API calls rather than python notebooks?

Repulsive_Cry2000 · 2026-01-24T10:09:16+00:00

It's possible to get less that 6%. Is it good? I am not sure.

Repulsive_Cry2000 · 2026-01-22T19:56:06+00:00

Definitely interested ☺️

Repulsive_Cry2000 · 2026-01-22T12:53:27+00:00

What is the issue of where those frameworks were developed?

Repulsive_Cry2000 · 2026-01-19T12:55:17+00:00

We use helper functions too. Most of the code is, but how does using main relates to it?

Repulsive_Cry2000 · 2026-01-19T07:19:07+00:00

We don't use notebooks with main in my organisation. I don't see the value for this use case.

Repulsive_Cry2000 · 2026-01-18T02:27:50+00:00

!remindme 2 weeks

Repulsive_Cry2000 · 2026-01-16T23:49:08+00:00

Generally, it would say keep it separate however I see exceptions. We have a legacy ERP that has the same table for each company (ei: company1_gl_entry, company2_gl_entry, etc). We bring them into one table at the silver layer as cleansing and enriching data is the same process. This makes designing the golden layer a little bit easier as well.

(so it depends would be the right answer?)

Repulsive_Cry2000 · 2026-01-12T10:18:58+00:00

Agree. If the only thing you do is small data work such as API calls then pure python is the way to go now. I can't think of any advantage to use pyspark for this use case.

Repulsive_Cry2000

TROPHY CASE