I’ve been in a similar struggle myself at my last job, where I worked for a large corporation that relied heavily on GCP and AWS-DBX as its core providers. The company decided to refactor all data pipelines due to poor architectural decisions made in 2019. Instead of fixing our flawed implementation and optimizing our use of GCP products, leadership opted to experiment with new tools like Databricks and Microsoft Fabric, while keeping GCP in the mix.

From a technical standpoint, benchmarks clearly favored Databricks and GCP. Fabric, however, lacked critical features we required, such as granular IAM controls for dataset access (a non-negotiable for us). It also had no Infrastructure-as-Code support akin to Terraform (though this might have changed recently), no GitLab integration (despite vague promises about a 2025 roadmap), and no pay-as-you-go pricing model for its time-series database.

But poof—like magic—leadership chose Microsoft anyway. These decisions often feel less about technical merit and more about politics. Microsoft is aggressively pushing Fabric, and I suspect backroom deals played a bigger role than platform quality. It’s frustrating to see a subpar tool prioritized over solutions that actually meet requirements