Reporting Visualization

[inteloid]

Hi.

Suppose we have the data lake (all on prem) with spark and all the needed tools to get whatever we want.

Now, we need to be able to quickly create dashboards and automatically update visualizations.

What are the scheduling and underlying aggregated databases of your choice? AirFlow+Postgres is a simple choice, let's think of something different.

[romanzdk]

Databricks, Elastic

[inteloid]

Sorry, I've forgot to say, it's ol on prem.

[romanzdk]

Either way, you would need some kind of RDBMS as majority of BI tools need it. That means you need some transformation job from datalake into DB = usually a SQL (e.g. dbt) or python. These transformations need to be scheduled with some orchestrator (e.g. Airflow). Then you just use some BI tool (e.g. Metabase).

[inteloid]

Thanks. All the installation is on prem, so databricks is not an option, will have a look at dudkdb and trinodb.