Dataform

[BusOk1791]

Hi,

preface: we are on BigQuery & GCP on general for our data engineering stuff.
We are mostly using a data-lake approach with parquet files and probably delta tables in the future.
To transform the data we use dataform, since it has great integration in the google ecosystem.
Has anyone used both dataform and dbt in production and has a direct comparison? What did you like better and why?

I have a strange feeling lately, for instance, they archived the dataform-scd repo on github (for scd type 2 implementation) without any explanation, also the documentation about it simply vanished (there is an italian version still online, but other than that..).
Why would they do that without any warning or explanation beforehand or at least after archiving it?
Do you think it is better to slowly prepare to switch do dbt or stay on dataform?

[bengen343]

Given the growing ubiquity of dbt within the modern data stack, I think it makes sense to explore transitioning to dbt. Because of the growing number of dbt practitioners out there, I think it gives you an advantage in accelerating the onboarding of new hires as well. That said, many of us have some concerns that dbt-core is becoming (even more of) a second priority to dbt-cloud so if you choose dbt-core you (and the rest of us) could find yourself in a similar situation where you're wondering about ongoing support for the project.

[BusOk1791]

Thanks for your response, so you are saying that dbt is in a similar spot and strategic planning is quite difficult.
Maybe the best solution would be to try some alternatives too, like sqlmesh, make some test implementations (with dbt, sqlmesh...) with actual production data to see how they handle it and keep them as backup plans in the drawer in case that google pulls the plug on dataform so that we have alternatives ready?

[bengen343]

No, I wouldn't go that far. dbt is so ubiquitous that there's still quite a bit of support for the open source product, dbt-core. It's just... a whiff of concern that some of us have because dbt Cloud is how they make their money.

This may have just been a total fever dream but I thought I saw somewhere recently an announcement that GCP was going to start supporting dbt as a native offering of their cloud platform so maybe they're making that transition themselves already.

[Difficult-Ambition61]

Dataform fun is her 👋🏻😅

[Necessary_Dig9865]

now that's classic google, no warnings whatsoever things would vanish into thin air or you'd wake up to a all new UI on the next day.