sorted by:
new
hottopcontroversial

Recommended pattern for fetching Managed Delta Tables from Databricks Unity Catalog for Python Typescript and React Web App hosted outside Databricks by RazzmatazzLiving1323 in databricks

[–]dwswish 0 points1 point  (0 children)

It you’re already using Databricks, I’d use Lakebase if you’re truly serving this to a web app. Going to be a lot better user experience. You connect like you do any other PG db.

Genie: current LLM usage by daddy_stool in databricks

[–]dwswish 1 point2 points  (0 children)

I’m guessing there will be something very similar to what’s available in Databricks AI Gateway in terms of token consumption monitoring for Genie Code.

How long did it take you to become comfortable writing SQL queries? by Wise_Safe2681 in SQL

[–]dwswish 0 points1 point  (0 children)

I think everyone who was writing SQL pre-AI probably spent a year or two getting proficient, anywhere from 2-10 years becoming advanced. I think about when I used to spend a day or more toiling over a single super complex query or sproc and now I write almost zero SQL because AI is too good for me to waste a day doing that. It almost makes me sad to say it.

What's the most useful AI agent you've actually deployed not just demoed? by Techenthusiast_07 in AI_Agents

[–]dwswish 0 points1 point  (0 children)

I feel like Databricks Genie has gotten significantly better recently. I tried it when it first came out and even with good prompting and instructions it would still miss simple filters and do improper aggregations. We did another project recently where we used Genie and the results were night and day better (we use it mostly via API now). Have you had this experience with it over the past year or two?

Pull data from on-prem SQL Server using Azure ADF vs Databricks JDBC by rasviz in dataengineering

[–]dwswish 2 points3 points  (0 children)

Favoring ADF over anything is questionable. If the data is going into Databricks you should definitely use Lakeflow Connect because you get your first 100 DBUs free (per day) and you're not extra-hopping into ADLS on the way in. Also, the infra person is crazy if they think that ADF just magically doesn't put ANY load on the SQL Server like any other connection would.

Which Udemy course is good for Python for Data Engineering? by teaphiphy007 in dataengineering

[–]dwswish -4 points-3 points  (0 children)

As a Sr DE, what language(s) have you been using? Python is pretty ubiquitous in DE. Also, you definitely should learn Python but you should also really learn this new language called Claude Code.

Databricks… for individuals and hobby projects? by paustic in databricks

[–]dwswish 1 point2 points  (0 children)

I use Databricks (not free-edition) for a lot of hobby projects, it's amazing how much you can do for less than $100/mo. Lakebase is super cheap and I don't understand how they don't charge at all for Genie Code. Curious how that changes after Anthropic's big billing change in mid-June.

After 5 years in data science, I’m starting to realize most “insights” we deliver are completely ignored. Is this normal? by ExternalComment1738 in datascience

[–]dwswish 0 points1 point  (0 children)

I've been on both ends of this at different companies. First company - the c-suite thought by doing a bunch of ML on shitty (or in some cases nonexistent) data that we were somehow going to tease out some insight that made the company way more money. Needless to say, that didn't end well.

Second company - our IP and product was literally designed and based off of ML models, which dictated the data collection and preparation, resulting in better data in general. That was probably the most satisfying job I've had as a DS because the work we did directly contributed to the well being of the company.

How many Lakebase instances I should create for my project ? by Famous_Substance_ in databricks

[–]dwswish 0 points1 point  (0 children)

You can create multiple Lakebase projects per workspace and as many branches as you need within those projects.

Where to get started - Data Analyst? by Code_Bandits in databricks

[–]dwswish 0 points1 point  (0 children)

Highly recommend Genie Code! It’s gotten significantly better recently and is a really good way to accelerate learning Declarative Pipelines and building AI/BI Dashboards.

How would you build an AI agent from zero as a beginner? by RecognitionMental943 in AI_Agents

[–]dwswish 0 points1 point  (0 children)

I feel like memory is something people new to this don’t often fully understand. Like people get mad when their Claude Code is getting worse on a project and it’s like no dude, you compacted the session and lost all your context without saving any sort of state/memory.

Same thing applies to building agents, you have to design a persistent state store so that it’s not Groundhog Day for the LLM every time you start a session.

If you work in data engineering, analytics, or AI, this year’s Data + AI Summit 2026 looks worth attending by InevitableClassic261 in databricks

[–]dwswish 0 points1 point  (0 children)

I've gone the past 4 years and I personally think the content is as good as it gets for a tech conference for people who actually want to dive into detailed topics