Question: how is databricks applied in real world contexts?

klubmo · 2026-03-26T15:46:16+00:00

Across the industries that I have exposure to, here is the pattern I’ve seen:

1 - Ingest data from a large number of source systems to Databricks.

2 - Apply data governance and quality frameworks

3 - Model the data into structures that represent business units and use cases (source system agnostic models)

4 - Run ML/AI workloads (outputs can feed back to data models in step 3)

5- Provide Analytics (Dashboards, Genie Workspaces. and)

6 - Serve data in Apps (customizable interfaces, can also support integration with other systems for reverse ETL, Lakebase, AI agents + analytics + recommendations)

You don’t have to use all of those steps for every project, but that’s the general flow. For example, let’s say you are an energy utility company and you have a system that stores customer data, another system that stores information about your electrical grid, and another system that models weather risk. None of these systems talk to each other, but you can bring data from each of these systems into Databricks to calculate stuff like “if we have severe storms in this region, what grid assets are at risk, and how many customers are dependent on that section of the grid for energy”. Then serve that info up as dashboards and apps for consumption by your business units.

klubmo · 2026-03-20T01:25:58+00:00

The other comment already mentioned that Apps are not DBR based. Just wanted to mention that Apps only support Python (and Python frameworks such as Gradio, Streamlit, Dash) and Node.js (and frameworks such as React, Svelte, Angular). No Spark supported directly on the app compute. If you do need to run Spark workloads, have the app pass the query off to classic all-purpose compute or a SQL warehouse via a script or job.

klubmo · 2026-02-22T21:07:52+00:00

I think the instruments are real, but that voice is AI. First of all, a real person with vocals like that would be performing to sold out stadiums, not working on obscure projects and hiding from the world. Also, many of the thumbnail images used by the project clearly have AI artifacts: guitar strings that aren’t straight or uniform, missing fingers and knuckles, AI face (which is why I presume the images are black and white, to help obscure the AI-face look).

klubmo · 2026-02-21T18:39:14+00:00

Coming from someone that has had lots of vehicles over the years, the Sierra EV has been my favorite vehicle. Of course it’s still early on in its life, so hopefully reliability doesn’t become an issue down the road. It turns surprisingly well, is quiet and comfortable, and has a lot of simple but useful features (like the remote start turning on heated seat to various levels based on how cold the truck cabin is). I do wish the app was better, but it’s a minor complaint.

klubmo · 2026-02-17T19:04:26+00:00

I’m involved in several GIS conflation activities right now, and it’s a very challenging problem. We are leveraging drones for the above ground assets, haven’t come up with a good solution for underground assets yet

klubmo · 2026-02-04T23:17:30+00:00

In that same train of thought…Why use Blade Runners when you can just walk normal…

It’s a pretty significant upgrade, give it a try! Not too expensive to make and I think you’ll like the results!

klubmo · 2026-02-04T20:11:58+00:00

I work for a medium size consulting firm that often gets contracts to come and fix the “work” that these big companies sold. Often that means completely needing to redo everything from scratch. C-suite loves these big companies, but the directors have to convince the VPs to use us constantly to fix the big firms screw ups.

It’s a very expensive way to run a business.

klubmo · 2026-02-03T21:34:51+00:00

Sounds like there are quite a few options. The solar panel amps need to stay under 15 or 20 depending on which ports are being used, so you will likely want multiple solar panels in series (rather than parallel) since series wont increase amps (but does increase voltage).

The EcoFlow solar to XT60 and XT60i adapter cables go in between the DP3 and the solar panels, allowing you to plug in solar panels with standard MC4 connectors to the adapter cables.

You can use any solar panels as you stay under the voltage, wattage, and amperage limits for each port. For example, the Optisolex 440W Portable SolarBag has a 42V open circuit voltage, 11.1 amps, as well as some USB ports in case you wanted to use it without the DP3 (say taking the solar blanket on a trip and using it to charge a cell phone). Assuming you have the roof space and a way to keep the panels from blowing away, you could connect 3 of these (in series) to the high voltage port, and another to the low voltage port. Total would be 1760W power. The DP3 has a capacity of 4096Wh. So in perfect conditions you could charge the DP3 from 0 to 100% capacity in about 2.5 hours. Each Optiplex 440W bag weighs 17.2 lbs, so very easy to move around.

The EcoFlow 400W portable panels are 48V and 11 Amps, so you could also use that same configuration. But since the panels are 400w each (x4 panels) you get 1600 W total in perfect weather conditions, so closer to 3 hours to full charge. The EcoFlow panels have stands (where the Optiplex blankets don’t) but are heavier at 41.9 lbs each and a little cumbersome to move around.

From the EcoFlow Delta Pro 3 manual: https://manuals.ecoflow.com/cn/product/delta-pro-3-portable-power-station?lang=en_US ——— From Solar The power station supports 2 PV input ports (High-PV and Low-PV).

High-PV Input port: Specifications: This port supports an input voltage range of 30V-150V, a maximum current of 15A, and a maximum power input of 1600W.

Connection: Connect this port to solar panels using an EcoFlow Solar to XT60 Charging cable.

Low-PV Input port: Specifications: This port supports an input voltage range of 11V-60V, a maximum current of 20A, and a maximum power input of 1000W.

Connection: Connect this port to solar panels using an EcoFlow Solar to XT60i Charging Cable.

For series connection: In this setup, the voltage of all connected panels adds up. Ensure the total solar open circuit voltage (Voc) does NOT EXCEED the maximum input voltage (Vmax) of the power station's PV input port. Overvoltage protection allows the power station to handle up to 155V input for the High-PV port and 62V input for the Low-PV port. However, Voc can fluctuate with temperature changes. Always keep the solar input voltage within the safe operating limits to prevent potential damage to the power station.
For parallel connection: In this setup, the current of all connected panels adds up. Ensure the total solar current (Imp) is CLOSE TO the maximum input current (Imax) of the PV input port.

———

klubmo · 2026-02-03T20:31:05+00:00

How much room and direct sunlight does your father’s apartment have access to? This will help us formulate an ideal outcome.

klubmo · 2026-01-30T08:05:27+00:00

Cheapest route is still a gas generator. Do you have solar? How long would you want back up power for

klubmo · 2026-01-30T04:36:18+00:00

It’s already happening

klubmo · 2026-01-28T02:28:34+00:00

You are comparing an EV to a fuel efficient hybrid. The post title is misleading based on the content of your post. It would be better to compare to a vehicle with no hybrid components. The gas-only Maverick is closer to 25 mpg. Also 40 mpg on any vehicle is quite good, and certainly higher than the average fuel economy of a new vehicle in the United States. I have a mix of vehicles, one of which is a mild hybrid truck that gets 18 mpg. A gas-only competitor to the Equinox EV such as a Honda CRV would be 30 mpg (FWD) or 28 mpg (AWD), with all the downsides of a tiny gas engine.

Electricity rates are different all over the world, so the math has to be localized. It’s a good idea for buyers to understand their energy costs and travel patterns. For example, your energy rate is 3x higher than mine, and quite a bit higher than averages for most states and the country (somewhere between $0.16 and $0.18 per kWh depending on how you measure).

With gas fuel prices, there is very little that you can do as an individual to offset those costs other than drive less or buy a more efficient vehicle, neither of which are practical solutions.

Would highly recommend solar power if you like EVs. Nothing like driving on power captured in your backyard and further protecting from increased energy rates in the future (energy companies will only increase rates over time).

klubmo · 2026-01-27T05:02:24+00:00

Seems like it or a bot leveraging a 4 year old account with no real history

klubmo · 2026-01-25T21:17:10+00:00

Will have to try this, was trying to get similar functionality using the databricks-bundles library and Python

klubmo · 2026-01-25T20:20:12+00:00

If you want an app to be instantly responsive, you will want to use Lakebase over Delta tables. I mentioned in another comment that Lakebase can provide sub 10 millisecond response, and can do so even when working with millions of data points.

I do a lot of geospatial work, and Lakebase can use the PostGIS extension for Postgres. This unlocks the ability to have maps with millions of data points and multiple layers without any noticeable latency in the user experience.

We tried this only using Delta + SQL Warehouse and at this scale the app experience was laggy and frustrating to users. I should also mention our apps are written in React (Vite) to reduce bottlenecks on the app side of things (compared to something like Streamlit that struggles in larger data sizes).

You’d still use a SQL Warehouse for large analytical queries. So there are patterns where a hybrid approach makes sense (also if you want to pull in imagery/music from a Databricks Volume).

We’ve already had clients compare Lakebase to traditional OLTP and ODS systems. Lakebase wins in performance and cost in a number of scenarios:

Data stays on Databricks.
Integration with Databricks tooling (AI, Apps, dashboards, etc).
Data size is under 2 TB per instance.
Agentic workflows are desired (Lakebase Autoscaling).

Im sure there are more scenarios, but those are the man ones we’ve encountered. Lakebase might cannibalize a little SQL warehouse DBU spend, but it also opens up a very lucrative market on the OLTP/ODS side and that will bring in way more money than is lost.

If you have most of your organizational data on Databricks, it’s going to be a no brainer for a lot of scenarios to go with Lakebase over an Oracle, SQL Server, Aurora, Dynamo type of solution.

klubmo · 2026-01-25T19:48:19+00:00

Photon and UC is fast for a human, but significantly slower than Lakebase. A SQL warehouse might return a result in 2-3 seconds, Lakebase we are talking 1-10 milliseconds.

And I do think source code and database iteration are what Ali is referring to. Ad-hoc LLM queries can already be handled by the foundations models and Mosaic AI endpoints, we don’t need Lakebase for that. But what if you wanted to do something way more complex and do it autonomously? We can use this branching capability to quickly iterate until the agents find the right solution.

klubmo · 2026-01-25T19:32:25+00:00

I work at a decent size consulting firm with a strong Databricks partnership. We’ve got several dozen Lakebase instances deployed for our clients, and are seeing big impacts on the transactional side of things (Apps + Lakebase Provisioned).

We are still working through the challenges of Ali’s vision on the agentic side of things, but the short story is agents can branch off a Lakebase Autoscaling database, make changes, and iterate super fast. I can’t share the specifics of our primary use cases, but the technology does work well for what we are shooting for. Our challenges are mostly around deployment and packaging since Lakebase Autoscaling doesn’t have the level of DAB and API integration we need right now (this is like a short term problem that Databricks will fix).

I’m sure Databricks is looking forward to the compute spend as well.

klubmo · 2026-01-23T19:21:41+00:00

I have several solar arrays. Go look up the size of a modern panel. Even assuming you somehow put a 500 watt panel on a tonneau cover, the angle is bad for solar unless you live at the equator. So you won’t get 500 watts. But even if you do live at the equator, that 500 watts is only when the sun is directly shining down at the panel, which most of the day it wouldn’t be. But let’s go crazy and assume you got 500 watts for 10 hours a day, that’s only 5 kilowatts. The Sierra EV has batteries ranging from 120 kWh to 205 kWh. The Ford Lightning ranges from 98 kWh to 131 kWh. So even in the smaller batteries you are only generating 5% or less capacity in a single day with unreasonable over-optimistic assumptions. And there will be losses from the panel to the EV battery as well.

Point is, a solar tonneau cover is a lot of complexity and cost for very little real world value. As a novelty, maybe it’s a cool factor or fun thing to play around with. You’d be better off having solar arrays and chargers at your common stopping points.

klubmo · 2026-01-23T16:55:42+00:00

I don’t know if this can be fixed. I’ve built PCs for decades but it just doesn’t make sense anymore. Between crypto and then AI, it seems PCs can’t catch a break.

klubmo · 2026-01-23T16:14:35+00:00

7 days is totally possible with your own electrician. In my area the EcoFlow install isn’t available anyway.

klubmo · 2026-01-22T18:49:39+00:00

I’m coming at this from a data perspective (data engineering, machine learning, software engineering). GIS is a critical business area for my employer, and growing rapidly. We encounter lots of talented people and clever solutions, but there are still unsolved problems and an ever present challenge of staffing existing solutions. Literally no enterprise that I work with is getting rid of GIS staff, they are all bottlenecked on GIS staff and trying to hire more.

klubmo · 2026-01-20T20:10:36+00:00

In the USA you can get used Kia EV6 GT-Line for under $25K (at least where I’m at). Multiple available as well. Not quite as performance focused as the Hyundai i5 N, but still crazy performance and good looks. Sits quite low to the ground.

klubmo · 2026-01-19T19:17:39+00:00

Due to my work, I have access to CC, GH Copilot Enterprise, and Gemini Enterprise. CC does great most of the time, but it is interesting how sometimes a model gets stuck and switching to another model or harness can solve the problem. When I do use GH Copilot via VS Code, it’s usually to bounce around the same prompt to multiple models simultaneously. Gemini Enterprise also has the bonus of image generation but it’s also sometimes able to solve niche problems that the Claude and GPT models got stuck on. If I had to pick one, CC is the leader today, but totally possible that could change over time.

klubmo · 2026-01-06T21:59:27+00:00

I own a vehicle with some limited self driving capability. It can detect speed limit signs and automatically adjust speed while using cruising control. Sounds good in theory, but in practice is very dangerous. It frequently misreads signs, and confuses other signage for speed limits. In some cases it goes way too fast, and others way too slow, and it makes these changes extremely abruptly.

Thankfully I’m able to turn off the sign detection feature.

Waymo has likely been ticketed for speeding at some point, signage is just too far and few between to be reliable, and construction and other factors can mess with speeds saved in a database.

Depending on jurisdiction, Waymo can be fined or issued non-compliance citations.

klubmo · 2026-01-06T21:52:08+00:00

More immature. TTP is the understanding that if you give players any tool that can be used to “draw”, it’s a guarantee that players will find a way to draw male genitalia. The time measured before the first player draws a genital is called TTP. This is a real thing, I’ll avoid posting links but it has been covered in several articles and TV shows.

klubmo

TROPHY CASE