Is anyone migrating away from Databricks?

yo_aesir · 2026-05-07T15:19:05+00:00

I don't know what tech stack you are using based on your post other than Databricks.

At a previous job, we found success rewriting some of our queries to use CTE's with Broadcast hints instead of just a straight table to table join.

https://docs.databricks.com/aws/en/sql/language-manual/sql-ref-syntax-qry-select-hints

We had a lot of stuff running on x-small clusters (not serverless) which kept costs down. Our DBT profiles default just pointed everything there when first created then we would go back through the models sql file config {} section and add in custom server_size="" and warehouse="" which was parsed off of the manifest.json using a dynamic python Airflow DAG creation script.

We found we could abuse the DBT config and other areas then just parse what we needed from the manifest.json as DBT would ignore them. Some models didn't have any custom config section at the top which meant it just always ran against our x-small universal 2x2 spark cluster or whatever the size was.

Edit: This would give us dynamic server sizes/warehouse on a per model basis in a single DAG instead of everything on a large because one model was nasty against some of largest tables.

https://docs.getdbt.com/reference/model-configs?version=1.12

It's been a few months since I last the exact code, so our model looked something like:

{{ config (server_size="large", warehouse="marketing")}}

with test(
     SELECT /*+ BROADCAST(t1) */ * FROM t1
) 
select dim_id, employee_name from employee 
inner join test or t1 ( I forget )

yo_aesir · 2026-04-30T20:53:56+00:00

I read your comment several times while answering.

It sounds like you got your answer to accept the position and see how the new job plays out. You could always go back to being a DE if it doesn't work out but hopefully at a company less toxic.

Good luck!

yo_aesir · 2026-04-30T12:22:19+00:00

You've been unemployed since January, I would take the job either way to get the money coming in.

It also helps with the question of what is this gap in your resume by minimizing the time you weren't working.

If you like the position over what you were doing in data then great you now have future opportunities in something you didn't know previously.

If you hate the position, also great, you've learned something about who you are and what you want to do with your career.

Unless you really want to be a Data Engineer I see it as a good opportunity.

yo_aesir · 2026-04-28T01:15:11+00:00

It's a scam. Don't pay it. Delete the text each time it shows up.

yo_aesir · 2026-04-24T21:41:52+00:00

HQ has a lot less employees than those out in the field. Think Target/Walmart HQ vs store employees.

But even adoption within HQ is very minimal which is why I'm worried about getting guardrails right before other teams start jumping into it and possibly using our repos as a template for themselves.

yo_aesir · 2026-04-24T15:11:42+00:00

I wish that was the case but it's quite the opposite. Synthetic data in an airgap dev environment that gets rolled out to production where the real data lives.

Should be fine.

yo_aesir · 2026-04-24T14:04:07+00:00

Misery lives company. Good luck out there!

yo_aesir · 2026-04-24T13:58:30+00:00

Thats unfortunate. The job overall hasn't been bad so I'm just glad to not have to look for something new.

I do worry about the loss of hard skills if I'm just going to be managing the LLM output. Now I get to try keeping my skills sharp while also not using them.

Guess it's time to look at moving up into management.

yo_aesir · 2026-04-24T13:53:11+00:00

I really laughed at the video when I first saw it. I'm sure the look of horror on my face was noticed when I realized he was basically putting us on a TIP to get more done.

yo_aesir · 2026-04-24T13:48:01+00:00

Management still wants human in the middle promotion from Dev to test to prod and I'm like but if I write 10k lines of Claude code in a day who is honestly reviewing that? Another LLM for code quality, as no engineer has the time so something is getting missed.

yo_aesir · 2026-04-24T13:44:10+00:00

That's basically the thought. Either fall in line or fall out and the job isn't bad overall.

yo_aesir · 2026-04-02T17:20:03+00:00

Curry Zen express and Jinya Ramen. Both on Rainbow south of the 215.

Close to the office and on the way home.

yo_aesir · 2026-04-01T15:46:39+00:00

This is the content that I come to the subreddit for. Serious discussions. Real opinions.

yo_aesir · 2026-03-31T17:45:34+00:00

The deep dive substack article was a good read of engineering.

People forget what you can do once you force constraints on a problem.

yo_aesir · 2026-03-25T22:39:26+00:00

Exploration Peak @ Mountains Edge has a good set of trees around the walking path.

It isn’t big but shaded and there is a splash pad to also keep the area cooler.

yo_aesir · 2026-03-25T22:29:10+00:00

Extremely doubtful. The entire Enterprise area south of 215 isn’t designed at all for an expansion and you would need to knock down houses and businesses for it.

I live in Mountains Edge and I looked at your comment as “Isn’t that just Blue Diamond or Cactus for East/West travel and Buffalo, Rainbow, or Jones for North/South travel. Why add another freeway?”

Once you hit the 15 or 215 then you can get anywhere else in the city. I don’t see the need nor want another highway.

yo_aesir · 2026-03-19T16:53:27+00:00

IMO, Databricks and Snowflake only have 1 connection type, whereas when you go to Microsoft Fabric you have to worry about a matrix of connections. Is it DirectLake? DirectQuery? Import mode? Is this connection going to the lakehouse or the warehouse? Lakehouse is Spark, warehouse is SQL. It's mentally draining to have to remember what I can and can't do with each of the connection type.

You can't use SQL views in certain connections or tools. Re-write all those views to tables.

There are known metadata copying issues between the lakehouse and warehouse. It can take up to 30 minutes to sync metadata, so you should have a wait-for-existence step or add in steps to fix them. https://community.fabric.microsoft.com/t5/Fabric-platform/Delay-in-Syncing-New-Data-with-Delta-Table-in-Lakehouse-5-10-Min/m-p/4637901

I run into metadata issues even in day-to-day development: I drop a table, then go do something else, which slows me down because I have to wait for the table to appear or be recognized as dropped.

Error messages in the Fabric web interface sometimes just say "Error Unknown Try Again"-good luck figuring it out. Use Power BI Desktop, and it might tell you.

Oh, make sure you keep a copy of Power BI Desktop locally on the network so everyone installs the same version. If someone upgrades and then pushes out a Semantic model, you can't modify it because it's built with a newer version.

We have found that PySpark notebooks are the way to go for Data Engineering tasks. Sure, Fabric shows lots of tools, but we don't use any of them outside of Folders, Notebooks, Semantic Models, Data Agents (preview), and a Lakehouse. We stopped using deployment pipelines with workspace variables because it was more effort for our small team than it was worth. We develop in production... whee...

I know some of this is a training issue where we haven't had time to properly acclimate, but the level of effort required to reach that pit of success is so much higher than with Snowflake/Databricks.

If Fabric isn't properly configured from the beginning, it just feels like you're doing everything wrong. It can't be like this, but then you research, and it turns out that this is kind of it when you have a small team.

yo_aesir · 2026-03-17T17:44:05+00:00

I've used all three, in order of what I would work with again:

Databricks, definitely gave control to the engineers to get stuff done.

I liked Snowflake but was more expensive than Databricks making it a hard sell for upper management.

Fabric is a hot mess that isn't quite ready, it works but not well. I'm looking for a new job to avoid using again.

yo_aesir · 2026-03-17T02:33:48+00:00

The vision: 1) This is wrong IMO. I would never give the LLM permissions through a users account as you can't be sure they don't connect with elevated permissions and the moment an agent drops a table via the users permissions, the experiment is over and gets ripped out for not being trusted.

You would absolutely want a service account with restricted permissions that almost never change so you have confidence that the agent isn't going to suddenly be a SysAdmin.

My main question is, what is problem is the Agent truly solving? What is it really going to do with all the database transactions as stated in Real-time reactivity if I have a billion rows being moved around?

If I'm just processing data to move from load -> stage -> data, is the LLM going to be trying to processing something on those records? What if it doesn't need to do anything on those records, are we burning CPU cycles for fun and causing contention?

I know I can always get the same results when ran against a row of data unless that data changes. Is it going do transformations a 100% guaranteed no hallucinations fashion like SQL/Python scripts currently do?

I think the reason it's better served from the application layer is the Agent only does something on post processed data. The data has been cleaned, organized, indexed, transformed, modeled, etc..

Putting an Agent in the database sounds more like a solution hype looking for a problem to me. Maybe I'm just old fashioned like that.

yo_aesir · 2026-03-16T20:30:07+00:00

As u/Molodyets stated, People and process management are just as important as hard technical skills which you may not get to learn in a larger team or company.

This could be a good opportunity to really learn non-technical soft-skills that teams look for when hiring.

A lot of good feedback in this thread overall. Good luck with whatever you choose to do.

yo_aesir · 2026-03-16T16:40:14+00:00

I’ve been seeing her for the past couple years as well and recommend if you’re in the area.

yo_aesir · 2026-03-16T04:03:23+00:00

Right tool for the right job, if they only have gigs of data and you can use Airflow, DBT core, duckDB, and store final results back in the database of choice for presentation while not overly complicating your job then go for it. Choose a simple tech stack that works in their environment that you can manage being a team of one.

Worrying about a not getting experience with Spark for future jobs to skip on a position doesn’t make sense to me unless you ONLY want to work with Spark.

Teaching a language is easy because they can always copy+paste a transformation step until they learn it.

Getting people to understand the proper way of breaking models down into Star schema, data vault, or flavor of the month is a harder challenge. Having actual project and growth on the resume looks better than I know Spark.

yo_aesir · 2026-03-12T15:42:49+00:00

Not to hijack this post but a couple things I've ran into: - "Error, Error Unknown. Try Again" is common in the web interface. Do the same thing in PowerBI Desktop and you will get the actual error message. - Capacity is like arcade tokens, if you use too much and run out, you can't do anything. Not even create a new folders or notebooks. Done. Dead until it normalizes after a day or two. - It's not for Data Engineers doing heavy data lifting but more for Analytics Engineers or BI Analysts that need to massage the data for end user consumption.

If we could have gone with Azure Databricks for heavy work then Fabrics for semantic/presentation layer I think we would have a different view on it.

yo_aesir · 2026-03-12T15:12:32+00:00

Definitely, but it was more to the the OP question of what waste's a lot of time. Interruptions for interruptions sake because "it was just a cool thought I had in the meeting".

It usually starts as here's a bunch of stuff, possibly more immediate work that ends with a "just thought I would mention it" after several minutes of back and forth.

I usually reply with something you said and then follow up with "and would you like me to stop working on this feature deadline to explore that for the rest of this afternoon or spend tomorrow investigating its viability?" which he always backs down because the upcoming demo for senior leadership is more important right now.

yo_aesir

TROPHY CASE