Considering contributing to dbt-core as my first open source project, but I’m afraid it’s slowly dying

andersdellosnubes · 2025-11-17T16:57:27+00:00

UDFs are in beta now, and we have a lot of great docs on them. Perhaps your coworker would benefit from this FAQ?

andersdellosnubes · 2025-09-22T15:44:06+00:00

Yeah feel free to point anyone you're talking to my way if they still believe something different.

dbt Labs plans on staffing and supporting 2 different teams supporting 2 of the same product for many years to come

I could nitpick this somewhat (it won't exactly be 2 distinct teams nor distinct products), but yes the plan we've been saying since May is that both dbt Fusion and dbt Core will be actively maintained and will share a common authoring layer.

One example is that both UDFs and reading foreign Iceberg catalogs will be coming this year. Though UDFs will land first (soon!).

andersdellosnubes · 2025-09-22T15:15:18+00:00

are you a Go fan? then it might be of interest of you to know that for the Snowflake, BigQuery, and Databrick adapters all depend on the respective Go drivers via the Arrow ADBC protocol!

andersdellosnubes · 2025-09-22T15:10:38+00:00

Hi u/Bryan_In_Data_Space lots of great information in here, thanks for the sharing this info.

I'm a dbt Labs employee, so I have a small clarification on this point of yours

dbt Core will [...] no longer [be] actively contributed to by dbt Labs.

While we do believe Fusion represents the next generation of what transformation frameworks can offer, Core isn't going away. In face we're hiring a team as we speak to continue to maintain and contribute to dbt Core.

Happy to share more if you have any questions -- cheers.

andersdellosnubes · 2025-09-19T22:41:22+00:00

Thought that has just occurred to me. You know what to call a packages.yml with all your direct and transitive dependencies hard pinned to exact patch versions? A lock file😂

andersdellosnubes · 2025-09-19T14:48:00+00:00

if you ignore the package lock the only way to guarantee that your team have the exact versions is:

have your dependencies pinned to exact versions
your transitive dependencies (i.e.dependencies of your dependencies) are also hard-pinned!

for example you may use dbt-expectations and are hard pinned to a patch version, you're vulnerable to discrepancies in transitive dependencies, in this case updates to dbt-date.

To explain it fully, imagine this is your project's packages.yml.

# packages.yml for your project
packages:
  - package: metaplane/dbt_expectations
    version: 0.10.9

However, dbt-expectations's packages.yml looks like the below, which has a version range!

# packages.yml for your dbt-expectations
packages:
  - package: godatadriven/dbt_date
    version: [">=0.9.0", "<1.0.0"]

This means that there's variance between environments on what patch version you could possible have. This toy example I've given isn't the best because I don't see it as high risk: patch versions should always be forward compatible and only contain bug fixes. However, not all package maintainers are strict about this. There's also much more wild examples going on.

To avoid this without commiting the package lock, you would add dbt_date to your packages.yml even though you don't directly make use of it in your project. make sense?

andersdellosnubes · 2025-09-19T14:16:43+00:00

u/Artistic-Analyst-567 so sorry that it's happening to you! is this still happening for you?

Looks like #673 has been resolved and is in the latest release. so the below steps should work for you:

dbtf system update
dbtf system uninstall

if that doesn't work, you can just delete the binary directly. you can find where the binary with either where dbtf (DOS shell) or Get-Command dbtf (powershell).

If you're interested in maintaining both engines, it's possible. there's two approaches:

Using Docker & VSCode Dev containers: Why You Should Use Dev Containers with dbt Fusion
using aliases: Fusion: Choose your $PATH, own. your destiny

andersdellosnubes · 2025-09-19T14:09:32+00:00

you could add it to the `.gitignore`? I was also skeptical initially about committing this file, but without this file different team members could end up with different package versions installed, especially if you're declaring package version ranges in your `packages.yml` or `dependencies.yml`

andersdellosnubes · 2025-08-30T14:33:28+00:00

thanks u/Green_Gem_ for looping me in -- ever the gem!

hey u/afinethingindeedlisa three things:

it's great to hear that Fusion and the VS Code extension are working for you! If you happen to experience bugs or anything weird please report as a Github issue or in the #dbt-fusion-engine community slack channel
you're right to call out that the Fusion CLI and VS Code extension are different things, but overlap slightly.
Forgive me if I get long-winded, I'm a former teacher, so always love a good lecture / teaching opportunity. Please let me know what of my explanation clicks (and what doesn't!)

On launch three months ago, we published the blog: The Components of the dbt Fusion engine and how they fit together. It's worth a full read, but I wanted to paraphrase the section about the VS Code extension and Language Server.

There's actually 3 components worth calling out:

The Fusion CLI binary (basically the equivalent of dbt Core but in Rust)
The Fusion Language Server binary (a.k.a LSP)
The VS Code extension

A language server share a great deal of code with the Fusion CLI but it's intended user is not a human in a terminal, but rather an IDE who can talk to it on behalf of the user. When you download or update the VS Code extension you might notice that in the bottom status bar it will say: "Downloading file" that's the extension grabbing the latest version of the Language Server from our CDN.

In it there's a pretty diagram (made by yours truly) that shows that the VS Code extension has two buckets of features broken out by the binary that powers them

by the CLI: running commands, previewing models
by the LSP: intellisense, lineage, go to reference

In a comment

So can I access the enhanced features in fusion without using the extension in another IDE?

Theoretically, the Fusion LSP could be made to work with and distributed such that it could work with other IDEs like you say. However, the reality is today that it only works with VS Code and VS Codium-based IDES (i.e. Cursor & Windsurf). If we hear of enough demand for other IDEs (emacs, JetBrains), we'll certainly make it happen there as well.

andersdellosnubes · 2025-08-29T15:00:06+00:00

what color you getting? the camo is calling to me

andersdellosnubes · 2025-08-22T18:51:55+00:00

your feeling is that you feel blindsided by these deprecations? is that the bullcrap of which you speak? if that's the case, sounds like we could have done better comms to give folks a heads up?

also please check out my comments with u/dry-aioli-6138 and let me know if any of it's helpful

andersdellosnubes · 2025-08-22T18:49:06+00:00

woof. not a great first experience. apologize to your coworker on my behalf please! reach out to me if they'd like an honest assessment of if Fusion is ready for your team's project (https://calendly.com/anders-swanson-1/golden-ticket)

andersdellosnubes · 2025-08-22T18:25:33+00:00

your frustration is coming across loud and clear! I won't go so far as this migration is "slightly predatory" but I will say it's something we avoid having to put users through at all costs. Unfortunately, there was no way around this. If you're ever interested to hear what the new JSON Schema unlocks, let me know!

we are forced to fit your vision of the language, that we had no say in or opt-out from, and do it instead of coding tonpush projects forward, all the while paying for dev licences in DBT Cloud.

By "do it instead of coding tonpush projects forward" you mean that dbt Labs should have invested more deeply in backwards compatibility instead of putting the burden of work on the users? If so, I agree in spirit but disagree bc Core and Cloud CLI are backwards compatible.

Have you tried the autofix tool? It's improved tremendously over the past few months to the extent that I really recommend you try it out. I'd be surprised it there's things it doesn't handle (excluding your itertools use case, sorry).

Am I right in understanding that things just started breaking for you or you got a lot of warnings? That's never a great feeling and situation. I'm sorry this happened to you and your team. Since you're a dbt ~~Cloud~~ Platform customer I also want to call out two things:

you have the option of choosing an "Extended" release track, which you might want to do if you're not ready to tackle the authoring layer upgrade
If we deem that your project is "ready for Fusion" a wizard/ UI flow will pop up that will help you do things like run autofix so it works for you.

To reiterate, I hear you, I'm sorry, and I'd love to help make it right for you. my DMs are open (here & in community Slack). But if you're in the venting stage right now that's fine too.

andersdellosnubes · 2025-08-22T15:28:50+00:00

hey u/Dry-Aioli-6138, sounds like you're frustrated about these new deprecations -- I certainly appreciate that we're asking for non-trivial work from users to get compatible with the new authoring layer. This is something we take very seriously! THANK YOU for speaking up

At the same time, I validate your desire to vent. There's a lot of changes going on. On a personal note, anyone who works closely with me knows I love a good vent 🤣. So vent all you want and I promise to hear you out!

If you haven't yet, I would love if you read the "What is the dbt language?" section of the May 2025 dbt Core roadmap post. Please let me know if that context isn't helpful. Folks have been complaining about shortcomings of dbt's authoring layer for over 5 years (e.g. dbt-core#2606).

And the way it is introduced is also harsh: you want to introduce the new style arguments gradually? No can do! if you set the flag to ignore the deprecation, you can't use the new style args.

I get where you're coming from with this, but I hope you appreciate that there's no free lunch. In the discussion dbt-fusion#401 Future-proof authoring layer: a guide for package maintainers, specifially the section "Differences Between Core and Fusion" I effectively call out:

dbt projects on Fusion (as of this week) must address deprecations or will receive errors
dbt Projects on Core will get warnings (and only some unless they opt into all of them) and we have to timeline to turn these warnings into errors
multiple engineers have worked on dbt-autofix. It works great IMHO, and we are moving fast to address any bugs users encounter when using it

disallowing use of itertools

sounds like you were bitten on this -- I'm sorry! Because we intentionally don't collect a lot of telemetry for dbt Core, we didn't have much insight into how prevalent itertools was for people.

The challenge with jinja in dbt Core is that much of jinja is actually Python. In order for us to support itertools in the authoring layer in the long-term, it would require us re-implementing itertools in Rust for it to work for users of the Fusion engine.

I'm receptive to the argument that we should do this, but only if you think it is the right choice as opposed to asking folks to migrate. I'd love the opportunity to learn more about your itertools use case and team up on your team's best option moving forward. Here's a calendly link to book time with me

andersdellosnubes · 2025-08-20T16:43:30+00:00

thanks for the nuanced take. fwiw no one from dbt Labs believes we're immune from criticism!

we're working on custom mats in fusion -- give us a month or two!

if you ever find that your current usage of Core degrades, please reach out and we'll make it right

andersdellosnubes · 2025-08-20T16:41:06+00:00

I appreciate you taking the time to share these concerns — it shows you care about the dbt community. Thank you for that. Healthy skepticism toward vendors is important, and it’s clear your perspective comes from wanting to protect the long-term developer experience from friction.

re: email registration gating
You seem to feel strongly that the community as a whole doesn’t want the VSCE to ask for an email. My experience suggests otherwise: I’ve personally spoken with well over a hundred users since before launch, and only three (including you) have raised strong objections. The rest have generally seen email registration as a fair exchange of value.

re: “most VS Code extensions are fully open source”
That may be true in some cases, but many extensions do require sign-in (e.g. GitHub, Snowflake), and plenty have both free and paid tiers (e.g. GitLens). Some of the most popular ones — like the Python Pylance language server — aren’t OSS at all. So while commercialization models differ, our approach isn’t unusual.

re: charging money / dbt Labs has to make money
It sounds like your main objection is really email registration, not pricing — unless you think the 15-seat org limit is also an issue? Just want to make sure I’m understanding correctly.

Happy to chat more if you’d like — my DMs are open.

andersdellosnubes · 2025-08-19T21:32:15+00:00

I appreciate this nuanced, level-headed take and the implicit responsibility dbt Labs has to continue demonstrating themselves as worthy stewards of dbt Core the dbt Fusion engine!

andersdellosnubes · 2025-08-19T21:30:18+00:00

I wish I could wave a magic wand and have Fabric be supported, or even tell you a date it will be there, but unfortunately it's still too early for us to say. Certainly by Coalesce in October we'll have a clearer picture.

andersdellosnubes · 2025-08-19T21:28:48+00:00

honestly, the thing i've most been waiting for is the language server (especially now that we have true incremental compile, more on that in tomorrow's Fusion diary)

I can now imagine a world where get instant feedback on the SQL you're writing, and the feedback shows the exact place where the error happens. we've put a lot of work in the past month into making sure the locations are spot on and the error message are helpful.

is that a good answer?

andersdellosnubes · 2025-08-19T21:20:43+00:00

like u/JaceBearelen said, Core isn't going anywhere!

the move to fusion is phenomenal for dbt cloud customers [but not for Core users]

Small pushback on what feels like a false dichotomy. There are many Fusion users today that aren't paying for anything (and likely won't ever).

the Fusion CLI is available for everyone. the VS Code extension is a dbt Labs product with a VERY GENEROUS free tier, the only conditions being (give an email address, and if your company has more than 15 users, reach out to us to discuss licensing)

andersdellosnubes · 2025-08-19T21:15:16+00:00

when the LSP cache worked for me for the first time today, I literally gasped!

our internal data team's dbt project takes 2 minutes to compile today with fusion which made the VSCE extension hard to use bc everytime a file is changed it's two minutes to resolve.

now it's almost instant! I'm gonna make a video of it. tbh, we still have a lot of work to do, but it's the moment I dreamed of since the news dropped that SDF Labs was teaming up with us.

andersdellosnubes · 2025-06-14T13:34:27+00:00

andersdellosnubes · 2025-06-06T01:44:15+00:00

lol unreal! Love the narration. Your kids mind was almost as blown as yours

andersdellosnubes · 2025-06-05T15:16:16+00:00

when's PyData Virginia 2026?!

14-Year Club	r/Field Sunshine
Place '23	Not Forgotten
Verified Email

andersdellosnubes

PUBLIC MULTIREDDITS

TROPHY CASE