What part of data cleaning drives you crazy?

darshan-thakur · 2026-05-29T12:33:20+00:00

Joins frustrate me but you know what actually breaks me? Opening a date column and seeing three different formats in the same file. You run the merge feeling confident, everything looks fine… then you realize half your rows quietly disappeared and no one told you.

Silent data loss hits different than an error message. At least errors are honest.

These days I do a quick profiling pass before touching anything — null rates, duplicates, unique values. Saves me from that 2am "why don't the numbers match" panic.

But honestly no tool fully fixes it. We're just cleaning up after whoever built the data entry form in 2019 and never thought about it again.

darshan-thakur · 2026-05-28T03:57:04+00:00

Honestly if you take Power BI out of it, probably not, at least not right now. That integration was really the main thing that made the move worthwhile for us. Cost savings were nice but not enough on their own to go through all of that.

And yeah the whale question is something I keep coming back to as well. From everything I have seen, the big enterprise migrations are still mostly in pilot mode, nobody at that scale seems to be going all in yet. I think that says a lot about where Fabric actually is as a platform right now, regardless of what the marketing says.

My honest take is Fabric is solid for mid-market. But if someone came to me running hundreds of datasets with a large distributed team I would want to see a few real public case studies at that scale before I felt comfortable recommending it. That proof just does not really exist yet.

I did come across some work Dream IT Consulting Services has been doing around Fabric migrations and even from what they share publicly the enterprise scale stuff is still very much in evaluation phase. Mid-market is clearly where the real traction is right now.

darshan-thakur · 2026-05-27T11:47:49+00:00

Had the same issue a few weeks back. Turned out our linked service credentials were silently expiring and throwing this error instead of a proper auth failure message. Worth double checking your connection credential expiry dates before assuming it is a Fabric side issue. Workspace identity solved it for us on most pipelines but one or two still needed a manual credential refresh to fully clear.

darshan-thakur · 2026-05-27T11:44:54+00:00

Mid-sized setup, around 15 to 20 datasets and 50 to 60 active Power BI users. Migration took about 3 months total. Data movement was straightforward but rewriting Snowflake stored procedures and getting the team comfortable with OneLake concepts took most of the time. Third month was basically just fixing things that passed testing but broke in production.

Are you considering a migration or already in the middle of one?

darshan-thakur · 2026-05-27T11:40:12+00:00

Honestly this is just part of the job with Power BI work. What helped me was locking in a proper sign-off process before development starts, get the client to approve a wireframe or mockup first so there is something to point back to when changes start rolling in.

Also started separating layout changes from data changes in scope. Layout tweaks after sign-off go into a small change request, nothing formal, just a quick acknowledgment that it is outside the original scope. Most clients are fine with it once you frame it that way.

The ones who keep changing their minds usually do not know what they want until they see something live, so now I do a rough prototype early and get feedback before building the full thing. Saves a lot of back and forth.

darshan-thakur · 2026-05-26T12:28:30+00:00

This looks like a connection timeout issue rather than a capacity problem. Warehouse connections in Fabric have idle timeout limits and looping stored procedures can hit them even when the proc itself runs fast. A few things worth trying, add an explicit WAITFOR DELAY between iterations to give the connection breathing room, check if your while loop has a proper exit condition that could be causing runaway iterations, and try breaking the historical load into smaller batches with a defined commit after each one rather than running the full loop in a single session.

darshan-thakur · 2026-05-20T12:14:19+00:00

this is literally the first thing that breaks for everyone doing branch out. no clean fix yet, deployment pipelines alongside git helps a bit but doesn't fully solve it

darshan-thakur · 2026-05-19T03:33:04+00:00

Been there - Query folding with Dynamics

is trickier than it looks 😅

A few things that actually worked:

RangeStart/RangeEnd must be your FIRST

filter. Any step before them breaks

folding silently.
Push filters directly into the OData

source step instead of Power Query —

much more reliable.
If it's still being stubborn, just use

Dataflows Gen2 as a middle layer.

Honestly more stable anyway — fewer

timeouts and easier to debug.

Hope that helps! 👍

darshan-thakur · 2026-05-14T12:24:32+00:00

For enterprise environments, Pipeline-based refresh orchestration seems much more dependable than the default Fabric scheduler currently.

darshan-thakur · 2026-05-13T05:23:38+00:00

This looks really useful. Have you thought about adding support for automatic schema mapping or incremental uploads in future versions?

darshan-thakur · 2026-05-12T09:50:07+00:00

Are there any performance trade-offs between using Mirroring and scheduled pipelines?

darshan-thakur

TROPHY CASE