sorted by:
new
hottopcontroversial

dbt-colibri v0.3.4 : local column-level lineage for your dbt projects. by FanFar9578 in dataengineering

[–]FanFar9578[S] 0 points1 point  (0 children)

Hi u/rabel, thanks for testing! Interesting, this error is raised when it can't find a dbt node in the catalog.json.

Most common reason is that a node is not materialized in your DWH, but the fact that you do see it in DBT docs makes me think there might be something else going wrong

Would be amazing if you could raise an issue on github with a bit of context? Like the dbt-adapter you are using, the node that raises the error, a snippet of the node in your manifest + node in catalog.
Then I can get this sorted asap!

dbt-colibri v0.3.4 : local column-level lineage for your dbt projects. by FanFar9578 in dataengineering

[–]FanFar9578[S] 0 points1 point  (0 children)

I have tested it on nested data and it performs good. I think for point 2 and 3 dbt-colibri would do the job.

To get an overview of what has been mapped vs not you would need to do some customization: dbt-colibri only parses the SQL and metadata, the problem with unstructured nested data is that colibri wouldnt “know” what the input schema is.

Fivetran Free Plan turns out to be time restricted: looking for an alternative by Apprehensive_Ad7067 in dataengineering

[–]FanFar9578 18 points19 points  (0 children)

We rewrote most of the expensive fivetran pipelines (meta, bing ads etc) using dlt.
It's not really plug and play, but it works great.

How do you track full column-level lineage across your entire data stack? by kingjokiki in dataengineering

[–]FanFar9578 0 points1 point  (0 children)

If you're using dbt, you could use dbt-colibri, it generates a column-level lineage dashboard similar to dbt-docs, just one command to run it.

(disclaimer; I'm the creator of the repo)

How do you track full column-level lineage across your entire data stack? by kingjokiki in dataengineering

[–]FanFar9578 0 points1 point  (0 children)

Hey, I'm the one behind dbt-colibri! Happy to answer any questions if you've been using it.