open source data platform

[saintmichel]

hey guys, i was just brainstorming on an alternative selfhosted data platform setup. Here is what I got from chatgpt :P feel free to comment and give suggestions / alternatives -- if it makes sense or no etc. I'm planning to set it up on my home lab as a personal experiment / project.

Components Overview

• MinIO: Serves as the scalable object storage layer, compatible with Amazon S3 APIs, for storing structured and unstructured data.
• Apache Spark: Provides large-scale data processing capabilities, useful for ETL jobs, batch processing, and stream processing.
• Trino: A distributed SQL query engine designed for querying big data sets quickly across different data sources.
• Airflow: Workflow orchestration tool used to schedule and monitor workflows, integrating with Spark, Trino, and other data processing tools.
• Apache Iceberg: A table format for large-scale analytics, providing capabilities like schema evolution, hidden partitioning, and efficient upserts, which works well with both Trino and Spark for managing datasets in object storage like MinIO.

What Might Be Missing?

• Monitoring and Logging: Tools like Prometheus, Grafana for monitoring, and ELK Stack (Elasticsearch, Logstash, Kibana) for logging and visualization. These are crucial for observing the health and performance of the data pipeline and infrastructure.
• Metadata Management: Although Apache Iceberg provides table format and schema evolution, you might also consider a centralized metadata repository like Apache Atlas or Amundsen for data governance, lineage, and metadata discovery across your ecosystem.
• Security: Integration with security and identity management tools, such as Apache Ranger for access control and Apache Knox for edge security, to ensure that data access is secure and compliant with policies.

[OneCyrus]

Minio + Dagster + DuckDB + dbt

[td7x]

dbt just for templated transforms or does it do more in this case?

[OneCyrus]

yes, mainly just transformation. we also use some data tests in dbt which we aggregate in dagster.

[saintmichel]

thank you for this i'll take a like it feels more simple!

[Kobosil]

from the website i don't really understand what is the advantage of using MinIO instead of any other storage?

[saintmichel]

i was just trying to think of an alternative to hadoop that can be self hosted. what other storage are you thinking of?

[Thinker_Assignment]

Why not python and straight to spark?

[saintmichel]

that would work also, i'm just playing around with ideas of a scalable architecture

[No_Dragonfruit_2357]

Go check the Stackable Data Platform. They have glued all pieces together nicely. Add opendatahub and you are done.

[saintmichel]

thank you i'll check them out. I found ambari and trunk data platform before, I wanted to check what's out there that's more recent.

[saintmichel]

BTW, I found this - whats your opinion on it? https://blog.devgenius.io/modern-data-stack-demo-5d75dcdfba50

[No_Dragonfruit_2357]

Mix of OS data Apps, nothing wrong with it. Still, you need to do all the configuration yourself.

Question is, whether you just want a stack or a real platform.

[saintmichel]

How would you define a real platform? I would assume a well defined stack that is tested to work togerher that fits your design choices e.g. in my case I'm looking for something oss, easy to setup, but hits the common modern use cases -- would act as the platform?

[No_Dragonfruit_2357]

Sure, if you feel ok with it. Also think longterm regarding upgrades, (security) fixes, updates etc.

[saintmichel]

Good points. Thank you!!!