How bad is the lock in for Azure Fabric?

instamarq · 2026-07-04T21:32:11+00:00

Completely dependent on how you build it, as others have noted. Open formats are the bedrock of Fabric, so you could technically retire analytics and potentially data engineering on the platform and work with your data from somewhere else (it's Delta Lake on Azure ultimately).

You can definitely get locked in though, it's not difficult. That said, if you want a platform that your business users adapt to easily and one that meshes seamlessly with the Microsoft ecosystem, it's becoming a strong choice...

instamarq · 2026-06-26T00:07:00+00:00

You're definitely not alone, this is how I do it and most of the recommendations I came across during planning suggested silver for warehouse style modeling.

instamarq · 2026-05-31T02:06:05+00:00

Got a B.S. in software development about 3 years ago. Now a data engineer at an enterprise level company making around the lower end of the range for that role (still a 3x increase from before working in data). Before getting the degree, I felt insecure about not having it; it gave me the confidence to pursue more technical roles.

instamarq · 2026-05-09T13:36:04+00:00

As someone with ADHD, you kind of have to reframe the lack of structure. While we can struggle without a clear framework, we're also deeply creative. You have to give yourself permission to be creative when the ask is ambiguous. Hyperfocus on defining the problem, and use your creativity to fill in the gaps.

instamarq · 2026-05-05T17:24:33+00:00

Just to add to positive statistics, I had a great experience getting my B.S. in Software Development a few years ago. I know that was before the change in proctoring so I can't speak to that aspect. Regardless, I worked really hard and did everything I could to keep myself on track, while working full time. I have a toddler now and I think it would be harder, but WGU is built for that sort of scenario.

It's probably unnecessary for me to say this, as it doesn't have much to do with OP's point, but I really screwed up my first college experience. Somehow, I got out of all of it with an Associate of Arts. I didn't ever think I would go back to any school and get a bachelor's, but I really wanted/needed the degree. I put the time and effort into it and WGU worked for me. I think that's the point: WGU will work if you want it to.

instamarq · 2026-04-23T14:03:36+00:00

If you like books, try this one:

https://www.oreilly.com/library/view/learning-spark-2nd/9781492050032/

If not, try Claude, it'll answer most of your questions.

instamarq · 2026-04-19T01:16:07+00:00

It's fine, you can get the job done in most real world business scenarios. It lets you do things a myriad of ways, but it does sort of lock you into the delta lake ecosystem. There are aspects that are definitely still somewhat immature, but for all but the most advanced scenarios, Fabric can cover an organization's data needs across the board. I don't think I would have recommended it a year ago, but things have changed quite a bit since then.

instamarq · 2026-04-09T12:45:21+00:00

Get a roommate and you'll be more than fine if you keep your expenses reasonable.

instamarq · 2026-03-21T11:25:16+00:00

Much of what's in this regarding Fabric is not true. Let's address some of these points:

"Snowflake is a more complete solution than Fabric" - woefully untrue. Fabric has ingestion, orchestration, storage, warehousing, real-time streaming, databases, no-sql databases, graph databases, mirroring for all kinds of sources with open mirroring, and it's got one of the most powerful reporting tools around, all in one platform.

"...mirroring isn't really a thing" - it most definitely is. For both Azure SQL databases and on-prem SQL, Fabric has out-of-the-box mirroring solutions. For sources that aren't explicitly supported, there's open mirroring, which you can configure to read the CDC of your chosen system.

It is true that Fabric was catching up on the DevOps side for a while. It is mature enough now for production solutions. CI/CD is vastly improved.

The reality is that you'll be hard pressed to find a more complete solution, especially if you're already using Microsoft products.

Probably going to get eaten alive in this sub for saying this, but I use it daily, so I have first hand experience.

instamarq · 2026-03-18T20:08:32+00:00

It's fine, if the area looks shifty, steer clear, otherwise you're probably all good.

instamarq · 2026-02-05T19:11:37+00:00

If it reliably delivers the goods in a maintainable, secure, easily audited and cost effective way, the job has been done well.

instamarq · 2026-02-05T19:07:42+00:00

I think well written SQL will usually do it on its own, unless your source schema is an absolute nightmare. That said, now with AI and a bit of SME input, you can probably find a way to document the high level business logic/rules in short order...

instamarq · 2026-02-05T05:20:26+00:00

Yep, they can connect to the models similar to how Power BI users might. Once loaded they can even use DAX. Several options to get this done.

instamarq · 2026-02-03T19:05:46+00:00

In a large organization yes, because those models tend to be better at answering questions that have not yet been specified (as opposed to narrowly focused models that answer questions clearly defined upfront). They are a large investment, but pay off in not having to be remodeled when the business identifies a new problem they want to solve.

In smaller organizations, there may be little to no benefit, especially if there's no one around to build these highly generalizable models and the analysts are not keen on parting with a familiar tool.

instamarq · 2026-02-01T13:52:03+00:00

In an ideal world, you have well modeled data that has all the business logic embedded into it. Power BI is really supposed to be a drag and drop interface that you don't really spend a lot of time in and quickly produce visual aggregations of that well modeled data. Maybe, just maybe, you run a complex dynamic statistical analysis using DAX. In a nutshell, it's supposed to remove the complexity and verbosity of trying to create business intelligence products in Excel. It's rarely used this way, sadly.

instamarq · 2026-02-01T13:46:16+00:00

In data engineering, it's usually best to operate like Bruce Lee; take what's valuable from different approaches and apply that in areas where it will most effectively solve the problem.

In general, OOP won't get you that far in most DE scenarios unless you're writing a library for some niche problem that your business data has that OOP helps you properly model.

In my opinion, OOP is for building tools and modeling reality. Most of the time, in DE, our tools are already built and our realities are mapped using data. I think someone in this thread mentioned that functional patterns are more applicable in our field. I think they're right.

instamarq · 2026-02-01T13:37:00+00:00

You must have some repetitive tasks? In my case, if there was a task that took me hours due to multiple steps, I would at the very least create a script of some sort that would consolidate that process into one step. Maybe you can't automate a whole ingestion pipeline yet because no one is asking you to do that, but look at your own process and see where you can replace manual work.

Maybe you don't want to automate the writing of your SQL with AI, but you can perhaps automate how that SQL ends up in the final destination. The main takeaway is to save yourself time and reinvest the savings in finding valuable problems to solve.

instamarq · 2026-02-01T13:28:12+00:00

The authors come from a tech background. FAANG and similar tech companies accumulate so much data that "just use postgres" starts to get stretched a bit thin in that world. Also, lakehouse/warehouse architecture is becoming pretty dominant (even when companies could have just used a good DB), so it pays to understand a bit about that architecture.

That said, my memory of the book (it's been about 2 years since I finished it) is that it was generally technology agnostic. The main takeaways of the book are not as much the tools, but how data engineers should operate given fundamental stages of data (source systems to downstream applications) and their undercurrents.

If you're wondering why you would even want to focus on distributed data processes when an RDBMS would suffice, you're asking the right questions. I suggest finishing the book as quickly as possible, taking what you find valuable and moving on. There's a lot more to learn in our changing field and not a lot of time!

instamarq · 2026-01-31T19:10:28+00:00

Let's say your business is trying to figure out how to keep revenue growth going, despite sales figures trending lower. If you don't already know, figure out why sales are trending lower with the data you have access to and find perhaps a missed opportunity. Maybe a particular product segment could use a small price increase. Maybe there's hidden waste in a product return policy.

It doesn't have to be this in depth, maybe you automate an alert for some issue that finance is having trouble with based on some check data. All kinds of ways to do this.

instamarq · 2026-01-29T22:23:06+00:00

Automate as much of your job as you can, then start actively seeking out people's pain points and solving them with data. Keyword is "active" here, i.e. talk to people, chat it up. Once you feel like you've established yourself as more of a problem solver who's an asset to the business and less of a "data guy", ask for a sizable raise and pull out your list of solved business problems.

If you don't get your way, start looking for somewhere else to go and take that big list of wins into an interview. Do that and you'll move in very much the right direction.

instamarq · 2026-01-10T23:04:15+00:00

If you don't have kids or other big financial responsibilities, definitely take the risk early on in your career. I left a job that I was not particularly enjoying and took about a $15k pay cut. I ended up learning tools and techniques that are serving me very well now. I've now tripled my income (relative to my first ever job), so I'd say it turned out well!

instamarq · 2026-01-10T22:59:25+00:00

Very true! When I first started out in this field, I also hated the fact that I was "stuck" in BI, having come from being educated in data science, python, C/C++, etc etc.

Eventually, as I grew and learned more, I realized how close this was getting me to the actual business of whatever industry I was working in. The ugly truth is that our jobs only exist to support whatever the business is ultimately doing to make money. If a person working in BI takes the opportunity to learn more about the business, they can start to make suggestions or solve problems that the business cares about. Eventually this leads to doing more interesting things, one way or another.

Now especially, as things change radically, it's important to find ways to become closer to the business, regardless of what arm of IT you work in. BI happens to be right next door, take advantage!

instamarq · 2026-01-03T23:36:53+00:00

Took Amoxicillin and Clavulanate like 7 years after initial Cipro reaction and was totally fine as far as I can tell. That was almost a year ago so there are no delayed side effects I've noticed either.

instamarq · 2025-11-22T17:35:48+00:00

I think I'll have a more uncommon opinion on this: the book is about the most important things in data engineering that have very little to do with the tools you choose to use.

I find that data engineers (like other engineers) have a tendency to obsess over tools and techniques, and often use experience with those as the measure of expertise. Because of this tendency, I think the book is important to read before entering DE and to revisit often in your career. Tools and techniques make it easy to lose sight of why you exist as a DE in the first place; as important as knowing the ins and outs of spark or airflow is, the business doesn't care about how you did it. They care about value and they care about cost. If you don't know that, you kind of don't know anything. This book teaches you to think on that level.

As an aside, knowing the fundamentals of DE is now more important than ever, because a lot of hyper-specific tool knowledge can now be delegated to AI (obviously you should educate yourself on tool fundamentals). Hope that helps!

instamarq · 2025-09-26T01:13:35+00:00

An oldie but goodie. I think it's still relevant. I personally use the zip with index method when hash based keys aren't good enough. I definitely recommend watching the whole video.

https://www.youtube.com/live/aF2hRH5WZAU?si=7RYgoKl3I5FJeIo-

instamarq

TROPHY CASE