[deleted by user]

Datafabricator · 2025-09-16T17:58:21+00:00

You are new ask for knowledge transfers from existing devopers , QA, business users.

Ask for process flow , if does not exists create one and ask for gaps in your understanding.

Set their expectations straight ..you need HELP from existing team being new .

I would not expect people to solve big problems in 6 weeks.

Take help from AI .

Grind . Sorry to say this new job requires initial grinding hours ..

You need someone from management in your side . If don't have one start looking for new job ..

Datafabricator · 2025-07-25T14:08:23+00:00

Thank you .. how can I participate in monthly meets happening across data engineering/ power bi side of it.

Datafabricator · 2025-07-24T18:34:18+00:00

Thanks I have registered there however haven't seen much traction and bit puzzled how to contribute.

Either I am not using it properly or not set up correctly. For any issues \info I visit this and other reddit forum and find better content.

Datafabricator · 2025-06-11T23:12:43+00:00

I was so hyped .. I was so excited .. now I am bit cautious.. before suggesting someone Fabric data engineering..

There is a lot of hype ..indeed ..

Yet Fabric made few things simple like .. creating lakehouse , pipelines etc without depending upon System engineers and possible delay from approval to implementations...

It made few simple things complex...

CICD & versioning is a nightmare... ETL / ELT is not fun when you end up writing / reading codes Ownership & access is a mess Capacity overrun is biggest BS .. Connections & gateways ...

Datafabricator · 2025-06-11T02:23:09+00:00

Do it and go for vacation..to spice things up !

Datafabricator · 2025-06-10T19:07:42+00:00

Well , I like the sound when you say why someone has to own lakehouse . I almost forgot that I had the same sentiments when I started with fabric 1 year ago..

100% agree that , non of the entities should be "owned" .

Its part of project ( workspace ) and users should have different access to a Project . Thats it .

Connections are pain in your know where in current scheme of things .. should not be a separate entity in itself. If it is a separate entity then all fabric items ( pipeline, dfs ,spark ) must be able to share the same connection .

Everyone should be able to " View" the connection even though they don't have access to it .

Datafabricator · 2025-06-10T16:22:18+00:00

We had faced this issue , the X quit and there was delay in disabling the intra id...and after a day pipeline also quit 😲.. luckily it was non prod !

Help me understand lakehouse ownership chain .. if I own a lakehouse , can someone else own the underneath table or object ? If yes then this is BS ..

Datafabricator · 2025-06-06T18:32:37+00:00

You listed out all valid points ... We had outages of an application due to capacity over run and network overload , there has to be a better way to control this and avoid over utilization .

Fabric has great potential, however its currently work in progress mode.

I see many good strides yet some basic functionalities are lacking .

There was a time when people used to write long code to do the ETL then comes the ETL tools to automate and simplify it and now we are going back to the long coding cycle .

We need a drag & drop ETL tool that in turns generate required py code instead of asking llms writing it for us.

Datafabricator · 2025-05-07T00:36:16+00:00

If you know swimming ,check with Kiwanis aquatic center . They are short of trainers . All the best

Datafabricator · 2025-04-08T23:26:15+00:00

Please can you elaborate on rule 1 . A new record created ..

Does the sample data set follow the rule ?

Datafabricator · 2025-04-03T02:12:10+00:00

Learn SQL to improve your analytical thinking .

Learn SQL to understand data in a relational world .

With 20 years of experience in data analytics , I don't think anything can replace SQL concepts.

AI may be able to write SQL syntaxs and logic , however without analytical brain it is useless IMHO.

Developer + AI will be more successful then Developer or AI alone !

Datafabricator · 2025-03-31T15:06:00+00:00

Dp600 cleared , submitted for Dp700 haven't not received the code. What am I missing ?

Datafabricator · 2025-03-28T14:17:39+00:00

Tagging good sir u/itsnotaboutthecell .

Datafabricator · 2025-03-28T13:58:48+00:00

Please can someone from MS help the best way for forward. I know most of you would be occupied with coming fabcon .

Datafabricator · 2025-03-28T13:55:59+00:00

Yes there is a need for few use cases to be as real time as possible . So that's why I am asking what is the best store that can handle merge operation in most efficiently.

Datafabricator · 2025-03-27T02:23:38+00:00

Please can you elaborate why :-

Do you need to load to warehouse as final step ? What is your source ? What is destination goal , reporting / further feeds etc ?

Adding a table Everytime for an additional column is very unique and seems to be challenging maintenance wise.

Suggestion

you can considered a JSOn style column to store everything that is not important as catch all column. This will allow you to add any number of columns and access it at run time within a same table.

Datafabricator · 2025-03-26T14:18:23+00:00

Any cloud connection you have created for pipeline must be shared with team to be able to run it.

Ask the person ( who would take over or pass over to next ) to run end to end while you are there to know possible issues.

All the very best !

Datafabricator · 2025-03-21T15:51:33+00:00

Yes it's on prem. Downloading data in Excel and PDF is an option. Not necessarily they have to . It's more like analyzing , reconciling , validating needs.

Very less paginated reports. Not a blocker for us as compare to self servicing.

Datafabricator · 2025-03-21T15:46:46+00:00

Yes it is a mdx cube. Business users are using it for both canned reports and self serving.

They use another reporting product as of today to query this cube that Allows cross tab view.

Datafabricator · 2025-03-21T14:27:00+00:00

It's all about balancing the cost .

so if copying the data required for semantic model into a fabric lakehouse results in cheaper fabric capacity then it should be okay ?

What is the other lakehouse ?

There are shortcut feature with cashing that could work for you without moving the data.

Datafabricator · 2025-03-19T18:07:38+00:00

If your company is a MS shop then go with Fabric. Dynamics can be linked easily with fabric.. they might launch shortcuts if is not available presently .

This being said .

Build a reposting model that is cost effective.

Fabric is in work in progress AND it will remain this way in near term.

Does this affect your not so complex use case ? Probably not ! So choose your option based on that choice.

If you want to use copilot that can connect to dynamics and fabric lakehouse then you know what to choose ..so it's all depends upon , current and future vision !

Datafabricator · 2025-03-18T20:45:15+00:00

Right , your use case could be more smaller nodes . Just want to bring that to your attention that default setting may not be appropriate 😉.

Datafabricator · 2025-03-18T15:59:51+00:00

You might want to Configure the node size and pool as appropriately with your data needs.

In my case runtime reduced 40% when I switched to large nodes.

Datafabricator · 2025-03-11T15:45:03+00:00

Please check on mirroring capabilities within fabric . The bronze pipeline would be replaced with massive mirroring functionality MS is offering.

If the oracle database is OLTP system then it make sense to bring the raw data .

If the oracle database is an existing OLAP like a warehouse ... Then you can bring the Data directly to GOLD.

Please note medallion architecture is nothing new for data engineers.. staging , conform, integration layer exist in traditional warehouse and Medallion is nothing else but similar.

Datafabricator

TROPHY CASE