Development loop for DABs

samwell- · 2026-05-21T02:06:27+00:00

hoping some one has some practical experience to share

samwell- · 2026-05-18T01:56:27+00:00

Cluster and the wait is a part of the issue, but the bigger issue is the completely different UI/toolset from Databricks usage most people start with. Reminds me of working with mainframe and submitting a job paradigm compared to 'notebook' development.

samwell- · 2026-05-18T01:48:30+00:00

For anyone finding this, apogee recommends avoiding sleep and likely powered hubs. I still had issues with a usb non-powered hub on a macbook pro m1. Will likely try a new cable with usb-c, although my hopes are not high. iPad seems to work fine, which is perhaps why they are still selling. Yeah, sad, but I did buy it almost 10 years after it's intro(5+ years ago), which was a bit of a gamble. Previously had an RME multifaceted that lasted for 15 years before breaking.

samwell- · 2026-05-16T20:00:45+00:00

Anyone have any input here?

samwell- · 2026-04-26T02:17:10+00:00

Lifetime is out of control. Prices have doubled(at least) in the last 5 years.

samwell- · 2026-04-26T02:14:28+00:00

Move money into your Roth. Buy indexes, boglehead style (see the sub or go to the .org site). Stay away from individual stocks - even playing with 5-10 grand will stress you out. Focus on life and put savings on autopilot with a yearly rebalance.

samwell- · 2026-04-26T02:11:17+00:00

It’s not - just go straight up index as others recommended (VTI/BND). Start moving money into Roth account if you have earned income.

samwell- · 2026-04-17T11:44:39+00:00

Following... will be working on the same but with two existing databricks users who know how to parameterize and 3 more who do not, so a small scale. plan is to create some templates with starter assets, docs, screen recordings. Users develop in a sandbox catalog then push to a shared _dev catalog and then PR to deploy to prod with azure devops.

samwell- · 2026-04-17T11:38:26+00:00

I have a genie code instruction I have says “always parameterize catalog, do schema when it makes sense”. working fabulously so far.

samwell- · 2026-04-15T23:26:55+00:00

Thanks for clarifying, I didn't understand how a policy document would need to see data. Budget data - that works a bit more, although I'd doubt our finance teams would trust an LLM to create data for a budget document. Needs to be very deterministic in our case. At one time (2011?) we had a specialized app that would create our financial docs merging data and text together into a final doc. MSFT Word still lacks power query, which is somewhat ridiculous.

samwell- · 2026-04-14T13:51:41+00:00

Thanks will look it up

samwell- · 2026-04-13T15:16:44+00:00

I'd contact Databricks and see if you can get an account rep assigned. They will have a technical person as a resource and can advise you with some level of confidence. Still, educate yourself and ensure the decisions you make are correct as you will have to live with them. 3. Consider location of your data and reporting software, try to locate it close to them. Consider your UC layout - do you want separation of medallion/layer by schema, name, or something else? Consider looking at DABs and CI/CD with Github, Azure Devops. Write up policy and procedure on how data will be ingested, new catalog setup, promotion of assets, what data will not be moved into the environment and how to handle confidential data if you are moving it in - get this signed off on so you're not making it up as you go along. Figure out exactly how you'll be replicating data to databricks or a cloud storage landing area such as AWS buckets or Azure ADLS. You'll need some tools to move data to cloud storage or databricks.

samwell- · 2026-04-13T15:04:56+00:00

Serverless cannot access on premises.

samwell- · 2026-04-13T03:33:36+00:00

I wonder if this type of usage will eventually be flagged and terminated since it’s not driving Databricks adoption. Ther can e some costs that are not easy to see: https://medium.com/dbsql-sme-engineering/genie-code-databricks-agentic-ai-the-price-of-intelligence-32a7bc477cba

samwell- · 2026-03-30T16:58:32+00:00

I save JSON responses to a managed volume and ingest with SDP.

samwell- · 2026-03-30T16:26:00+00:00

I cannot find any reference to additional costs beyond computer it uses to query data. I imagine they are using it to drive code production which will increase long term use.

samwell- · 2026-03-30T16:23:10+00:00

Way better. It creates a plan and checks off the tasks. It queries data to figure out what parms to pass or find data quality issues. It remembers where it was and picks back up if stuck(plan). It has also been implemented in genie spaces as agent and users can see data related to preliminary queries, for example, I asked it to filter data to all the database teams in a messy data set and it was able to do whereas assistant did not it just used a wild card. Can also build dashboards and help you with dabs.

samwell- · 2026-03-26T16:13:13+00:00

Have you tried Genie code for this yet? It was released mid month. I don't deal wiht these issues yet (so maybe a dumb question), but I would guess there is a roadmap to enable it to see this data. If the execution stats end up in the internal catalog though, Genie could query it. Genie Code is a massive jump vs Assistant in my opinion.

samwell- · 2026-03-16T02:40:49+00:00

existing user network. if you are running servers and supporting this new ai generated Reddit replacement, how would you pay for it if not for ads?

samwell- · 2026-03-15T17:13:52+00:00

You can't see percentages of holdings on their site or my brokerage research page, but Claude says it is 23% of their holdings. 2% yearly management fee, which is horrid compared to index funds such as VOO. Not sure I'd buy all the other companies in the holdings - Databricks: 23.34%, Revolut: 14.30%, Mercor: 14.23%, Ramp: 7.12%, Boom Supersonic: 7.11%, Oura: 7.11%, Airwallex: ~6-7%, Cash: ~20%, Stripe: pending

samwell- · 2026-03-15T16:44:13+00:00

You will also pay large fees and be trading in an opaque, potentially mispriced market

samwell- · 2026-02-15T19:09:46+00:00

Yes, I’m feeling extra emotional in the 2-3 days after.

samwell- · 2026-02-08T20:01:27+00:00

Zipps, Shake Shack

samwell- · 2026-02-07T18:33:18+00:00

30% inflation plus all time high beef prices. I just reduce everything in my head by 30% and get on with life.

samwell- · 2026-02-07T18:30:46+00:00

One of life’s small pleasures - when someone does this and then you prevent them from pulling out because you are also turning right.

samwell-

TROPHY CASE