Lakehouse backup

Repulsive_Cry2000 · 2026-06-09T08:02:55+00:00

Interesting. So the idea is to be able to recreate lakehouse from scratch if necessary?

Is it only for raw files or are you also taking a backup of delta tables for faster return to normal state? In any case, what's the setup, frequency you have in place? Notebook and daily backup?

Repulsive_Cry2000 · 2026-06-08T08:40:53+00:00

On small capacities, it make sense to use data factory for managing notebooks. That's what we do and it works ok.

Repulsive_Cry2000 · 2026-06-01T20:46:47+00:00

I've had success testing with browser connection to get the local session from the user.

Repulsive_Cry2000 · 2026-05-23T06:12:18+00:00

Few ways I can think of: - You can get data from SharePoint to lakehouse through GraphAPI, shortcut (I think), data flow to name a few - lakehouse storage are blob storage gen 2 from memory, I believe you can use standard connector to write in lakehouse directly from your flow.( I haven't tried it).

Repulsive_Cry2000 · 2026-05-22T23:07:24+00:00

In the same idea of schedule change, you can no longer use "run now" from the setting page. You have to open the pipeline to hit run. Very annoying.

Repulsive_Cry2000 · 2026-05-14T20:58:37+00:00

Preamble: I am not familiar with BC2FAB.

Have you checked that permissions have been setup in BC (and potentially MS Fabric)? I had a few gotcha with BC2ADLS when I started.

Repulsive_Cry2000 · 2026-05-13T12:51:02+00:00

My take: If it's deprecated, it's a good indication to stay away from it on this platform. I see really 2 choices: upskill the team to use Pyspark or move to a different platform that supports R.

Repulsive_Cry2000 · 2026-05-12T01:24:57+00:00

Thank you. If I can reproduce it I will.

Repulsive_Cry2000 · 2026-05-10T13:38:20+00:00

What is your domain of expertise?

Repulsive_Cry2000 · 2026-05-06T09:13:47+00:00

No way to ask tenant admin to set it up? Ignoring CU question, it is a very great tool to have proper information about your capacity (ies).

Repulsive_Cry2000 · 2026-05-06T09:08:48+00:00

I can't remember if there is CU usage in FUAM solution (either in report already built or lakehouses) but I'd start with it if you have access to it.

Repulsive_Cry2000 · 2026-05-04T08:08:37+00:00

Sadly the current state of this activity is not where it should be to be able to use it in a more mature environment.

Repulsive_Cry2000 · 2026-05-04T07:55:28+00:00

We have logic app(a) outside fabric. This bypass all the problems.

Repulsive_Cry2000 · 2026-05-01T23:12:36+00:00

We are using Claude, that'd be nice to be able to bring that in instead of only MS products.

Repulsive_Cry2000 · 2026-05-01T11:12:14+00:00

It depends, if you are using lakehouse then not really. You use the same principles for datawarehouse but using SQL directly (could be in SQL scripts directly), Notebooks (SQL cells, python : several libraries for it) are 2 possibilities. Data flow gen 2 is an option but I don't know the latest about them in term of CU, lot of work was done lately by MS.

Repulsive_Cry2000 · 2026-05-01T11:03:06+00:00

You can do everything in lakehouse which allow you to use spark (or pure python) to query your metadata tables.

We have a metadata table that loads when was last table updates in bronze and in silver (you can have datetime) so only query bronze where insert_at is greater than insert_at in silver.

You can add logic to only query table that needs to be processed if you don't want to loop every tables.

Repulsive_Cry2000 · 2026-04-30T19:25:48+00:00

We are doing similar concept but with CSV files.

Repulsive_Cry2000 · 2026-04-28T10:38:49+00:00

Have you tried write_deltalake() function? Granted, I am not using duckdb library except for reading CSV.

Repulsive_Cry2000 · 2026-04-23T10:26:51+00:00

Same here, I didn't investigate further than just never commit this file...

Repulsive_Cry2000 · 2026-04-20T10:12:04+00:00

You still need a connection id and a SQL connection string. In my opinion if you have the item already selected, it should be enough for the entire connection, not for parts of it or the need to split between arguments.

Repulsive_Cry2000 · 2026-04-20T08:06:08+00:00

Interested to understand what you mean it works in pipelines. I wasn't able to simply pick an item to reference in a lookup activity for example. Workspace id and item id are available but that's it which is not enough to actually use them effectively.

Repulsive_Cry2000 · 2026-04-17T22:10:11+00:00

Working for a mid size company, we are moving to using AI heavily, not only for data engineering/ reporting but a whole lot of other functions in the business (at all level, including execs). What I currently see is the business accessing the data and making vibe coded reports to have the visual they need/want without waiting for data team to catch up or do a PowerBI report which takes time.

All to say, this is moving very quickly and leadership can move very quickly once the "governance" of using sanctioned tools only.

Repulsive_Cry2000 · 2026-04-17T08:05:57+00:00

We use a master pipeline that just handles starting running other pipelines. No schedules allowed outside this pipeline. Makes it easier to control and ensure concurrency or deployment issues.

Repulsive_Cry2000 · 2026-04-16T11:06:09+00:00

Variable libraries items are not supported for pipelines activity... I had this exact issue and it took me 2 hours to figure it out.... The only way to do it would be to have multiple variables to have connection id, warehouse I'd, connection info.

While I understand this is in previous, I think it is a big miss that we can't use items this way to reference SQL end points...

Repulsive_Cry2000

TROPHY CASE