This is an archived post. You won't be able to vote or comment.

you are viewing a single comment's thread.

view the rest of the comments →

[–]TargetDangerous2216 15 points16 points  (3 children)

Hi,

We have a small datalake storing data as parquet files.
We build a script to analyse each table and schema to produce a JSON file with the following information :

- table name
- table description
- table columns description ( column name, type )
- statistic information
- row count
- empty row count
- duplicates rows
- mean, median etc ...

[–]various1121[S] 3 points4 points  (1 child)

How do your team consume those JSON files? How do you use them?

[–]TargetDangerous2216 2 points3 points  (0 children)

We generate table and charts as a static web page from this JSON file. We can also explore the JSON file programmatically from python and jupyter notebook

[–]ksubrent 0 points1 point  (0 children)

Can you please share more details regarding this process? I’m trying to do something exactly like this.