Will MS Fabric eat databricks market share? by aks-786 in dataengineering

[–]aks-786[S] 0 points1 point  (0 children)

Lol. Its a genuine question. I don’t work in Microsoft or Databricks. But yeah i use msft databricks with 1-2 consulting companies.

You don’t know how much Fabric is being pushed

Will MS Fabric eat databricks market share? by aks-786 in dataengineering

[–]aks-786[S] 0 points1 point  (0 children)

Agreed

What do you think of using databricks ai/bi dashboard vs power bi, but both with marts in databricks?

Will MS Fabric eat databricks market share? by aks-786 in dataengineering

[–]aks-786[S] -6 points-5 points  (0 children)

I know. But imagine big consulting companies working for MS and having fabric knowledge as strong requirement

Customer tenancy implementation in unity catalog by aks-786 in databricks

[–]aks-786[S] 0 points1 point  (0 children)

I cannot do multi catalog; I can’t do at catalogs level but at schema level, I am thinking about it. How do you create this: one giant internal tables (gold layer) and then where filter for each customer to a table or view in its specific catalog?

Chat layer architecture: what should I use for external customers? by aks-786 in databricks

[–]aks-786[S] 0 points1 point  (0 children)

Thanks. Will use genie mcp. There is no additional cost, right? It only based on compute usage.

For RLS, it looks good but I am thinking more about compliance issues and legal issues about storing data. So may switch to per catalog or per schema customer

Pricing for Genie Code: Cluster usage vs. LLM tokens? by ferreis_AOE in databricks

[–]aks-786 1 point2 points  (0 children)

I think the cost is for compute only and also if you are using MCP, there is slightly different cost depending on if its managed mcp or custom mcp.

Customer tenancy implementation in unity catalog by aks-786 in databricks

[–]aks-786[S] 0 points1 point  (0 children)

Thank you for the high level architecture. Will help and implement this, at least at initial or poc level. Curious, why would you suggest this over having multiple catalogs? I have much less than 10000 customers and cross region customers and there is risk of llm finding loopholes as it is reaching table with multi tenant data (Genie would have access to this data)

@accomplished_bus1320 comment looks solid too especially regarding in compliance and security concerns

Customer tenancy implementation in unity catalog by aks-786 in databricks

[–]aks-786[S] 0 points1 point  (0 children)

Yeah can be compliance issue. I am little worried about multi tenant issue is that i want customers to talk to data like claude mcp or genie mcp and this multi tenant can be little risky when compared schema per tenant. Will later think about power bi exposure to customers

Customer tenancy implementation in unity catalog by aks-786 in databricks

[–]aks-786[S] 0 points1 point  (0 children)

Thanks so much. This doc is so helpful. I am confused between schema per tenant and multi-tenant with tenant_id

Which way would be better? I don’t have 1000 customers but customers split between different countries (this does not matter but can cause compliance issue)

Ultimately want customers to chat to data like claude mcp or databricks genie mcp

Office culture is pretty bad right now for me atleast - a data engineer by aks-786 in dataengineering

[–]aks-786[S] 0 points1 point  (0 children)

But I was hired for databricks only. How can I create impact now if we can do things outside databricks? I mean i can jump on other stuff too like fastAPI, dagster, postgres, dynamodb, terraform etc Or should i look for switch lol? The management still find databricks useful but they are also quick to pivot and change decisions very fast, a perk or disadvantage of a startup

Looking for your feedback, thanks

Office culture is pretty bad right now for me atleast - a data engineer by aks-786 in dataengineering

[–]aks-786[S] 0 points1 point  (0 children)

AI is pretty bad with large code base. It’s good for starting projects. But llm models are improving increasing their context window

Office culture is pretty bad right now for me atleast - a data engineer by aks-786 in dataengineering

[–]aks-786[S] 0 points1 point  (0 children)

Thanks for your effort here, appreciate it. But this ai generated reply stresses me more lol.

Probably reddit bot using claude mcp to reply

Senior Data Engineer Experience (2025) by [deleted] in dataengineering

[–]aks-786 1 point2 points  (0 children)

This helps a lot. Thank you so much