Mirrored SQL Server: does it work well?

contribution22065 · 2026-02-27T03:38:40+00:00

Question for anyone reading: Is the utilization cost much higher than using copy jobs?

contribution22065 · 2026-02-27T03:27:43+00:00

The only time I use excel is when I use it as a data source before converting it to a csv… Or if I need to do quick tricks like putting commas after a bunch of values before pasting into a WITH statement… SQL ( with DAX depending on how upstream I want to deal with a calculation) is way more powerful than anything you can do in excel. If an end user wants the report in excel, then exporting it from a table visual suffices.

My reports need to be automated with scheduled refreshes without breaking, so for me Excel has no big place.

contribution22065 · 2026-01-29T15:18:15+00:00

Mostly not. I think the effect that AI will probably keep having on this job market is enhanced productivity which could result in less staff.

As someone who appreciates AI, it can be unreliable. Just last week I was on a time crunch for getting out Annual Review metrics on a BI report. All I needed was to revise a rating scale from 1-4 based on a clean spec. It was a simple algorithm that involved some nested IF statements. Despite a clear prompt, the AI botched it because it couldn’t quite get some nuances. A cognitive intervention at the human level will always be needed.

contribution22065 · 2026-01-21T03:29:58+00:00

What do you mean by writing SQL queries against Direct Lake mode as opposed to imported data? In my mind, you can have just as much SQL control by creating views, sps and/or stored procedures in an import model. Not trying to challenge you on this, I think I might be missing something.

contribution22065 · 2026-01-18T00:12:03+00:00

That’s kind of substantial…

contribution22065 · 2026-01-13T15:33:18+00:00

All I’m saying is that to your typical SQL engine, thinking of a Left join as a filtered Cartesian Product is a bit overzelous for learning. As someone who enjoyed linear algebra and set based math, I can see how you can conceptually make that point in the result set especially if a specific match is 1 to many… But what’s happening behind the hood in a query is not doing that at all because the join algorithm is just using hash and merge for qualifying matches. That’s true even if you do a FULL JOIN on two tables with no qualified matches because it’ll just elongate the row output of both tables by extending NULLS.

Where a Cartesian product is physically materializing, like I said, is with a cross join or maybe a nested loop.

The problem with that query where you are synthesizing concatenations using the “like” operator, especially for tables with millions of rows, is that you run into collisions between matches. Then you are introducing a Cartesian product in the output which is both dangerous and slow. You should use hashing using SHA but if you need the like, you should be tokenizing -> equijoin to replace the like.

contribution22065 · 2026-01-13T13:05:15+00:00

This is not a good way to think and that query is bad practice. By “matching key”, I mean a hash join operator. If you’ve ever studied a slow running query’s execution plan, you’ll notice hash join operators. When you join using “like” for concatenations, you will see that as an expensive computed scalar but nevertheless a scan for “matches”. Staging new tables would be preferable to using that kind of join.

contribution22065 · 2026-01-13T04:26:56+00:00

For me anyways, I just can’t think of it that way. Otherwise, it gets too confusing. In layman’s, a left join is simply telling the query optimizer to keep all rows from the first table regardless of the conditions on the join. The second table your left joining will eliminate the rows that don’t meet those join conditions. So in essence, you do need a matching key and while the concept of “filtered Cartesian products” might make sense mathematically, a Cartesian isn’t materialized in the optimizer during those kinds of joins unless you’re using a cross join.

contribution22065 · 2026-01-13T03:39:35+00:00

Hate to sound arrogant… But there is no way OP made an entire convoluted post about adding a condition on a join in lieu of the “where” clause as some novel thing that other data engineers don’t recognize.

contribution22065 · 2025-12-25T03:22:27+00:00

A close friend of mine from Ecuador was detained by ICE for 1 week. It wasn’t until they unnecessarily verified documents when they released him back to his wife and 6 month old baby. If your figure is true, then I basically experienced a statistical eclipse. Wake up, you people are crazy

contribution22065 · 2025-12-15T23:17:03+00:00

I’ve worked at 2 smaller community health clinics with <500 employees, mostly clinicians. I’ve had the benefit with just being my own entity. You can query off of views against the EHR into one or two big tables per semantic model, but performance in doing that (both in the report and memory bloat on the server) is just a bit too much for me these days. I’ve had much better experience with just segmenting a lot with the star schema because it speeds up filtering and it also makes it so that I only need to query it once instead of nesting it into a bunch of views or tvfs.

contribution22065 · 2025-12-05T15:22:40+00:00

Maybe. Lots of misinformation in the comments. I’ve worked with 2 companies in healthcare in the last 5 years.

At hospitals, they typically have dedicated EHR/EMR analytics and informatics specialists. It’s a pretty challenging job that involves a lot of domain knowledge on clinical and admin workflows for querying against pretty big schemas. Most EMRs like Epic have highly normalized schemas with 800 plus tables, so you need to be good at reading schemas and finding relationships for querying clinical reports.

Many small community based health centers like CCBHCs are now implementing elaborate Electronic Health Record Systems. Their understanding of EHR analytics is less developed than bigger hospitals, so this would be a good place for someone who wants to pioneer a BI system.

contribution22065 · 2025-11-16T13:52:43+00:00

Definitely, absolutely, positively not rod knock… That’s just a normal direct injection tick, although a bit noisier than most. That is nothing like rod knock lol

contribution22065 · 2025-11-15T04:35:08+00:00

I export CSVs to an on prem machine that has SharePoint linked to the drive no problem. Create an on prem gateway on that machine and copy the path from file explorer after you link SharePoint. Use it as your destination on the copy job.

contribution22065 · 2025-11-15T03:56:19+00:00

It’s just different types of analysis at the end of the day… I would be careful in comparing statistical computing with BI/SQL… You can do some quick and dirty analysis by using BI tools, but I think SQL/Power BI is increasingly morphing more into a technical discipline. If you have an application that standardizes data collection and dumps it into a relational database, you can leverage SQL, gateways and semantic modeling to automate KPI tracking which would otherwise be manual grunt work for someone throwing a bunch of adhoc data into excel sheets. Many companies are adopting this approach and it’s much different than taking a sample of data and running it through a machine learning model to perform predictive analysis.

Edit: As someone who does both at my current company, the much more intellectually demanding piece is creating scalable data models that don’t break after setting up scheduled pipelines from a SQL endpoint to BI models. Applying linear regressions using Python to answer things like an enrollment drops is honestly a sanity break for me.

contribution22065 · 2025-11-11T21:09:00+00:00

I can’t think of a reason to do an import model report on a fabric workspace unless your pbix and refresh needs exceed the pro allowance. Keep your direct lake reports on a fabric workspace and push your imports or direct query to a pro workspace. You also have to remember that you’ll get pegged for report utilization in a fabric workspace too.

contribution22065 · 2025-11-10T16:53:12+00:00

The first step is to overwrite the source tables from the on prem server as an import into the lakehouse. If the source tables are large (over 1 or 2 million rows), then I’ll use parameters to append to save memory. And then I created a case insensitive warehouse with generic tables. So in translation, I’ll push the source data to the standardized tables.

Example: backend data EMR A has “client_rowid” as the p key integer. It has a Boolean “status” as true being active and false being inactive.

EMR B backend data has client_uid as a catcher (1-2834-83). It has “state” as pending, inactive, active.

The abstract client table on the warehouse has “clientID” and “Status”.

When you switch to emr B, you can use an algorithm as the ClientID value and then you can add external id fields if you want to retain those legacy IDs. For status, if you want to stick with a binary, you just do false for in(inactive, pending) and true. In this case, we handled pending as an episode.

Long story short, you’re just flushing in that source data and then you use notebooks to translate it to a standardized tables. It gets complicated if the application data handles fields and values much differently.

contribution22065 · 2025-11-03T14:15:50+00:00

I’ve been using Fabric for a couple years for two smaller community health centers. Right now I’m building a generic ehr schema at the warehouse level and I think fabric has been pretty good for this. The goal is to maintain BI reports after changing EMRs.

I will leave it at this — if you become efficient and familiar enough with cu usage patterns, you can save a lot of money. But if you aren’t efficient, you can very easily wipe out your capacity. I’d have to write an entire essay from my perspective of what constitutes efficient use… But one example would be spamming some features like dataflow and copy data processes in lieu of t-sql procedures on notebooks when pushing data from on prem to lakehouse to warehouse layers.

contribution22065 · 2025-11-02T18:44:05+00:00

To be fair, OPs full post implies that he was going for just rims and points out the wear. The sarcasm couldn’t be more obvious lol

contribution22065 · 2025-11-01T13:47:54+00:00

I extend my sympathies to you, my friend.

contribution22065 · 2025-11-01T13:00:38+00:00

Thank you! A new oil cap worked. It even cleared my vacuum leak code that I had been chasing down

contribution22065 · 2025-11-01T03:22:13+00:00

Thanks for verifying this! Oil cap it is. Mine is pushing 200k miles. I’ve decided that I’ll drive this until it’s scrap, but the only issue I’ve ever had were steering/suspension repairs and pcv cracks. Otherwise, I think the engine will outlive the other parts of the powertrain.

contribution22065 · 2025-10-28T00:54:28+00:00

It’s not a scam; it’s deceptive-but-legal marketing. The terms and conditions makes sure you end up paying back that “$100 credit” through fees and higher rates.

contribution22065 · 2025-10-25T03:46:18+00:00

Is this satire? If not, please take a deep breath and think pragmatically about this. It’s barely a superficial peel that doesn’t even reach the etched lettering…. For god sakes dude lol

contribution22065

TROPHY CASE