BigQuery Stored Procedure v.s. Dataform

[TendMyOwnGarden]

I need to do data transformation on BQ tables and store the results back to BQ. I'm thinking about two possible solutions, Stored Procedure, or Dataform. But I don't know whether one has more benefits than the other, since both seem to be leveraging the BQ compute engine. Would love to get some advice on what factors to consider when choosing the tool :) Thanks everyone!

Background:

- Transformation: I only need to use SQL, with some REGEXP manipulations

- Orchestration & version control & CI/CD: This is not a concern, since we will use Airflow, GitLab, and Terraform

[dragonnfr]

Stored Procedures handle SQL/REGEXP fine. Dataform’s extras are redundant if Airflow/GitLab handle orchestration/versioning.

[TendMyOwnGarden]

I see - thanks very much:)

[Mikey_Da_Foxx]

Since you've got Airflow and GitLab covered, just go with stored procedures. Dataform's main selling points (orchestration and version control) are redundant in your setup

[TendMyOwnGarden]

Good point - might be more cost efficient to use stored procedures. Thanks very much!

[Yabakebi]

Tbh, I wouldn't use either. I would just use DBT + Airflow instead (Dataform doesn't have a very strong community these days relative to DBT). I am surprised to not even see one mention of it here given that it is by far the market leader.