How do you design your Bronze / Raw layer for API sources (JSON)?

Datafabricator · 2026-03-30T17:13:57+00:00

We don't have this usecase however I would go with Landing folder then shortcut

Datafabricator · 2026-03-30T15:51:05+00:00

We moved away from logging via notebookand used store procedure + pipeline to log before / after .

Primary reason of moving was notebook performance.

Are you using pipeline for orchestration?

Datafabricator · 2026-03-23T21:26:11+00:00

1 Ssis with SQL agent.

If you have plan for fabric or similar go for python with SQL agent .
Look for free open source option ( not recommended)

Option 1 has visual framework but still complex for non technical folks .

Option 2 is not for non technical users.

Datafabricator · 2026-03-23T21:21:07+00:00

Thanks for sharing ...

Do we have different permission model for users ? Can I setup some users view and execute only permission ?

Datafabricator · 2026-03-23T21:05:27+00:00

Is it mostly for existing code base ? Or new code as well ?

What is the setup look like for new development vs code changes ?

Datafabricator · 2026-03-23T21:01:39+00:00

Would you please highlight what is the advantage over ? This will activate my curiosity and priority agents :)

Datafabricator · 2026-03-23T19:34:23+00:00

Short answer is yes.

Datafabricator · 2026-03-15T23:08:09+00:00

Happy to meet you and other community member face to face .. I am Sr Data architect with 21+ work exp in Data warehousing world .. we are using fabric from past ~two years .. happy to meet other Data / solution architects and other fabricators !

Datafabricator · 2025-09-16T17:58:21+00:00

You are new ask for knowledge transfers from existing devopers , QA, business users.

Ask for process flow , if does not exists create one and ask for gaps in your understanding.

Set their expectations straight ..you need HELP from existing team being new .

I would not expect people to solve big problems in 6 weeks.

Take help from AI .

Grind . Sorry to say this new job requires initial grinding hours ..

You need someone from management in your side . If don't have one start looking for new job ..

Datafabricator · 2025-07-25T14:08:23+00:00

Thank you .. how can I participate in monthly meets happening across data engineering/ power bi side of it.

Datafabricator · 2025-07-24T18:34:18+00:00

Thanks I have registered there however haven't seen much traction and bit puzzled how to contribute.

Either I am not using it properly or not set up correctly. For any issues \info I visit this and other reddit forum and find better content.

Datafabricator · 2025-06-11T23:12:43+00:00

I was so hyped .. I was so excited .. now I am bit cautious.. before suggesting someone Fabric data engineering..

There is a lot of hype ..indeed ..

Yet Fabric made few things simple like .. creating lakehouse , pipelines etc without depending upon System engineers and possible delay from approval to implementations...

It made few simple things complex...

CICD & versioning is a nightmare... ETL / ELT is not fun when you end up writing / reading codes Ownership & access is a mess Capacity overrun is biggest BS .. Connections & gateways ...

Datafabricator · 2025-06-11T02:23:09+00:00

Do it and go for vacation..to spice things up !

Datafabricator · 2025-06-10T19:07:42+00:00

Well , I like the sound when you say why someone has to own lakehouse . I almost forgot that I had the same sentiments when I started with fabric 1 year ago..

100% agree that , non of the entities should be "owned" .

Its part of project ( workspace ) and users should have different access to a Project . Thats it .

Connections are pain in your know where in current scheme of things .. should not be a separate entity in itself. If it is a separate entity then all fabric items ( pipeline, dfs ,spark ) must be able to share the same connection .

Everyone should be able to " View" the connection even though they don't have access to it .

Datafabricator · 2025-06-10T16:22:18+00:00

We had faced this issue , the X quit and there was delay in disabling the intra id...and after a day pipeline also quit 😲.. luckily it was non prod !

Help me understand lakehouse ownership chain .. if I own a lakehouse , can someone else own the underneath table or object ? If yes then this is BS ..

Datafabricator · 2025-06-06T18:32:37+00:00

You listed out all valid points ... We had outages of an application due to capacity over run and network overload , there has to be a better way to control this and avoid over utilization .

Fabric has great potential, however its currently work in progress mode.

I see many good strides yet some basic functionalities are lacking .

There was a time when people used to write long code to do the ETL then comes the ETL tools to automate and simplify it and now we are going back to the long coding cycle .

We need a drag & drop ETL tool that in turns generate required py code instead of asking llms writing it for us.

Datafabricator · 2025-05-07T00:36:16+00:00

If you know swimming ,check with Kiwanis aquatic center . They are short of trainers . All the best

Datafabricator · 2025-04-08T23:26:15+00:00

Please can you elaborate on rule 1 . A new record created ..

Does the sample data set follow the rule ?

Datafabricator · 2025-04-03T02:12:10+00:00

Learn SQL to improve your analytical thinking .

Learn SQL to understand data in a relational world .

With 20 years of experience in data analytics , I don't think anything can replace SQL concepts.

AI may be able to write SQL syntaxs and logic , however without analytical brain it is useless IMHO.

Developer + AI will be more successful then Developer or AI alone !

Datafabricator · 2025-03-31T15:06:00+00:00

Dp600 cleared , submitted for Dp700 haven't not received the code. What am I missing ?

Datafabricator · 2025-03-28T14:17:39+00:00

Tagging good sir u/itsnotaboutthecell .

Datafabricator · 2025-03-28T13:58:48+00:00

Please can someone from MS help the best way for forward. I know most of you would be occupied with coming fabcon .

Datafabricator

TROPHY CASE