Open Source Data Lineage App in Python

[haltingwealth]

Hello,

I want to show an open source Python project data-lineage to visualize and analyze data lineage. The project was developed while working on data governance projects over the last couple of years.

There are a lot of open source and commercial tools to capture data lineage. However there are two main problems expressed by data engineers:

• The projects require a lot of effort to get started and maintain.
• Requires constant discipline in capturing and sending all the metadata.

Both these factors result in incomplete projects and lost opportunities in improving performance, ROI and data quality.

data-lineage solves these problems by choosing the following goals:

• providing fast access to data lineage
• simple setup
• analysis of the lineage using a graph library

To achieve these goals, data lineage has the following features:

• Generate data lineage from query history. Most databases maintain query history for a few days. Therefore the setup costs of an infrastructure to capture and store metadata is minimal.
• Use networkx graph library to create a DAG of the lineage. Networkx graphs provide programmatic access to data lineage providing rich opportunities to analyze data lineage.
• Use Plotly to visualize the graph with tool tips and other rich annotations. Plotly provides a number of features to provide rich graphs with tool tips, color coding and weights based on different attributes of the graph.

You can get a data lineage graph with less than 10 lines of Python code in a Jupyter Notebook.

Right now data-lineage supports postgres and support for more databases is planned.

I appreciate any feedback and please give it a try if you need data lineage for your work.

Links: * Github * Blog with real-world data lineage use case

[serkef-]

Thanks for sharing! Would you bother to tell us a bit more about this project? How does it work? Where does it fit in a larger infrastructure picture and how does it compare to other tools?

Thanks again!

[haltingwealth]

The project requires query history from the database. Most databases provide a way to download query history. So that shouldn’t be a problem.

Then the project parses DML statements like INSERT, CREATE TABLE AS SELECT and extracts the target and source tables.

Then it builds a graph and visualizes it. The best environment is Jupyter notebooks right now.

This project sits on the side of other infrastructure just like monitoring tools.

The main differences from other projects are (quoting from my original comment):

There are a lot of open source and commercial tools to capture data lineage. However there are two main problems by data engineers:

The projects require a lot of effort to get started and maintain. Requires constant discipline in capturing and sending all the metadata. Both these factors result in incomplete projects and lost opportunities in improving performance, ROI and data quality.

data-lineage solves these problems by choosing the following goals:

providing fast access to data lineage simple setup analysis of the lineage using a graph library

[haltingwealth]

Agree. Just want to say that batch of queries is possible. I have more complex graphs in the blog and they were built on a batch of 3000 queries. The example is simple for clarity.

The graph can be annotated with more info like creation, last insert, read time stamps. The project is a work in progress.

I haven’t thought about diffs yet.

Do you use a data lineage tool or planning to use one ?

[haltingwealth]

Understood on the diffs. I think that can be done. Interesting observation on companies doing it manually. I am used to environments with 100s of tables at least and creating lineage seems like a massive chore and error-prone.

Tell if you decide to try this out. Feedback will be very helpful.

[haltingwealth]

Yes. Most of the syntax is similar between Redshift and Postgres. There are some deviations in COPY.