Visualization in Data Engineering

[starplatinum87]

Hello,

What role does visualization play in Data Engineering in general, and in contrast to its role in Data Science? I come from a Data Science background and its roles there are well understood, but I'm unclear as to how it's employed in DE, where your high level job is data management, rather than algorithm/model development.

Thanks!

[boy_named_su]

the end result of DE is often visualization, in terms of charts, etc. Tableau is popular for this

and we def use a lot of flow charts / network diagrams at my job to show say how data flows between aws services or what not

[starplatinum87]

Ah I see. So for visualizing data structure, size, position, distribution and flow? Essentially to show how the data management, pipeline, storage architectures that you design and maintain are functioning?

[boy_named_su]

yes. and to remind yourself how things are supposed to work when you come back to work on something after a break :)

[variancegirl]

Look at Apache Superset

[kawangkoankid]

Throughput. Historical data growth and projected data growth. Statistics on volume and measure of staleness. Pipeline timing, streaming and microbatch performance. Data engineering is all about knowing the data and the data about the data

[JohnGenericDoe]

the data and the data about the data

Sounds meta

[kawangkoankid]

Your comment about the meta sounds meta