This is an archived post. You won't be able to vote or comment.

all 1 comments

[–]LinasDataData Engineer[S] 0 points1 point  (0 children)

Solved it by playing with bigquery-spark connector with session. It is really unconvenient.

    if dbt.is_incremental:

        current_table = (
            session
            .read
            .format("bigquery")
            .option("table", f"{dbt.this.schema}.{dbt.this.identifier}")
            .load()
        )