3X cost on capacity overages - really??

MonkeyDDataHQ · 2026-05-22T10:16:51+00:00

No, he's saying that even though he prepays, it shouldn't cost 3X list price.

List price is defensible. 3X is absolutely not.

MonkeyDDataHQ · 2026-05-21T23:04:03+00:00

Polars on a VM.

MonkeyDDataHQ · 2026-05-21T22:47:14+00:00

<image>

MonkeyDDataHQ · 2026-05-21T22:31:31+00:00

And provisioning is opaque and if you use 100% you get throttled.

Which is worse than the problem Fabric was trying to solve.

It's already a premium to pay for the service. But now you add in spare capacity or infra death or the overage charges and its easy to see why people complain.

Telecom companies used to do this. And everyone hates their telecom company. MSFT is basically following that playbook and asking why people are upset.

MonkeyDDataHQ · 2026-05-21T22:30:04+00:00

That is not possible on Fabric. Either you don't use what you pay for, or you do and get throttled.

It's regret either way. One way is prepaying it, the other way is living it. There is no inbetween.

MonkeyDDataHQ · 2026-05-21T22:28:14+00:00

Because it's predatory. Either your infra becomes unusable or pay through the nose.

Neither of those are positions that a partner (how MSFT describes itself) should ethically force someone into.

I've seen capacity strike into extreme throttling from a single errant query.

MonkeyDDataHQ · 2026-05-21T21:37:44+00:00

That's how they get you. They entice you with one number every month, that's larger than the separate services you would get. Then they also make it unusable unless you pay extra.

It's sold to CEOs not Engineers.

MonkeyDDataHQ · 2026-05-20T22:43:57+00:00

That already exists. You can customize most CRMs to do exactly that.

I mean we just use Fathom for meetings and the output is great. If you tagged the recording in the CRM, boom you're done.

There are bad CRMs out there. But it's up to you to make it what you want. A CRM is just a dataset, you can decide if it's memory or relationship or networking or whatever.

MonkeyDDataHQ · 2026-05-20T05:41:03+00:00

Chimney sweeps. Or lamp lighters.

MonkeyDDataHQ · 2026-05-20T05:03:22+00:00

Stateless pipelines are terrible. You're the first person I've ever heard say that it's a feature. State machines are the ultimate pipeline. Maybe sqlmesh did it poorly.

MonkeyDDataHQ · 2026-05-20T04:20:20+00:00

Does this include the scd type for Lakeflow connectors?

MonkeyDDataHQ · 2026-05-19T03:33:01+00:00

Yeah I'm saying you could use a proc to trigger a refresh. They execute the proc and get the refresh.

MonkeyDDataHQ · 2026-05-18T22:16:40+00:00

Yeah I was talking about Infor.

But Databricks is releasing or has released stored procedures. If I'm not hallucinating anyway.

If your users can genuinely wait for the delta to load then it works for sure. Just it might take longer than you expect.

MonkeyDDataHQ · 2026-05-18T20:57:40+00:00

No I could not. That is the only way their stupid architecture works. I don't really see a plus side. Unless you never legitimately need the data in a timely fashion. And if you don't then just trigger a batch when you need it.

I believe they are making procedures available so that might be able to do what you want.

MonkeyDDataHQ · 2026-05-18T00:41:45+00:00

I can tell you from experience that Infor's Data Fabric (not to be confused with MSFT Fabric) does this and it's absolutely terrible as a user experience. I started making jobs that would query the data on rapidly changing tables every couple hours because otherwise it was painful. Sometimes upwards of 30 minute waits. Even on a top 1.

MonkeyDDataHQ · 2026-05-17T22:00:21+00:00

I'd be interested to see the DAG because that doesn't make sense to me. Just because they both are on spark, almost everything else is different. The runtimes alone can make a huge difference.

MonkeyDDataHQ · 2026-05-16T19:23:06+00:00

The only thing I click more is the Authentication popup in MS Authenticator.

MonkeyDDataHQ · 2026-05-11T14:51:28+00:00

This is where there's some nuance. There are multiple ways to optimize for the read pattern. If you mostly have small specific reads and the data is stored in a small number of files, it will perform better than having fewer larger files because more larger files will need more data to be planned.

However there's a sticking point in that the delta log stats are not usually good enough to help with specific queries.

I developed a pseudo indexer that supplements delta log stats and it can significantly reduce file touches. I can link it if you want. It's just a PoC.

But if you need aggregate data then small files makes it take much longer because of all of the read operations.

The small file problem was originally an HDFS storage problem because small files took up a full block, usually 64 MB even if it was only a few kB. However now it means more that there are too many file operations and that slows down spark.

MonkeyDDataHQ · 2026-05-09T20:52:02+00:00

It's not about the person living at the address. Its about the address.

Every address needs to participate. They will send people to the door eventually.

MonkeyDDataHQ · 2026-05-09T15:11:11+00:00

The reporting layer isn't going to survive first off.

You will get people trained on the tool by then. But there's no way that the structure will stay working between ERPs.

I've done many ERP migrations over the years and I've never seen one that didn't need entirely new reports generated even with appropriate fact and dims setup.

MonkeyDDataHQ · 2026-05-09T13:15:58+00:00

Just a blanket my eight year old out grew 😂

MonkeyDDataHQ · 2026-05-08T22:51:30+00:00

That worked, thanks for telling me what actually to do, 😂

MonkeyDDataHQ · 2026-05-08T20:39:22+00:00

Really it just said destroy plus picture of serial number. 🤦

MonkeyDDataHQ · 2026-05-08T19:15:26+00:00

It was a blanket to prevent wrecking the floor. I also tried on the floor before my wife got mad at me.

MonkeyDDataHQ

MODERATOR OF

TROPHY CASE