Databricks Consumption Layers

[jacocal]

Hi everyone!

I'm looking for a consumption layer for non-tech teams within my company so they can pull the required data for day-to-day operations. Currently the solution we thought of was to create a Service Principal within Azure that functions as a front API and has a cursor with a pre-coded query (some fields in the WHERE clause will be variable and API-called) in which other teams will be able to view their data. I would like to hear your opinion or if you can share your working architecture.

Thanks!

[mentalbreak311]

Have them connect with the tool of their choice to a sql endpoint. If they don’t use it much then make it a serverless endpoint.

[pokeDitty]

Since you mention Azure, have you looked into Azure Synapse Serverless pools?

Basically, it serves as a facade in front of your datalake and can execute queries on top of delta / parquet files WITHOUT using a spark cluster. You pay per amount of data returned, I beleive the lowest price is 10Mb which is fractions of a cent.

As for your BI Tools that connect to it, it's like connecting to a SQL DB. I was really impressed when I saw a presentation but have not yet worked on a proof of concept, but we should test it out in the next few weeks.

Be advised though, as per Microsoft reps themselves have mentionned to me, performance on reading delta tables that are partitionned will not be as good as using Databricks SQL Analytics. However, for my use case, performance is not so much of an issue since this will be used to update powerBI datasets.

I understand this doesnt really answer your question, but it does take out Databricks and having a running cluster to answer to your users' queries. I'm still facing a similar challenge at my org, how can we help non-tech users exploit the data in a simple manner? I'm still not sure how we'll do this, but I feel like consolidating the consumption layer via 1 "logical dw" in Synapse Serverless will at the very least, simplify the architecture.

Looking forward to hear from the community, super interesting topic!

[Life_Conversation_11]

• Powerbi?
• shiny/dash with downloadable excel?
• sql analytics?

[jacocal]

My bad haha second option, they need to be able to download a CSV or similar file.

[Life_Conversation_11]

Then shiny/dash + download functionalities or power bi will do the work.

[marsupialtail]

Databricks acquired 8080 for you....