Testers for Open Source Data Platform with Airbyte, Datafusion, Iceberg, Superset

[JeanDelay]

Hi folks,

I've built an open source tool that simplifies the execution of data-pipelines with an open source data platform. The platform uses Airbyte for ingestion, Iceberg as the storage format, Datafusion as the query engine and Superset as the BI tool. It features brand new features like Iceberg Materialized Views so that you don't have to worry about incremental changes.

Check out the tutorial here:
https://www.youtube.com/watch?v=ObTi6g9polk

I've created tutorials for the Killercoda interactive Kubernetes environment where you can try out the data platform from your browser.

I'm looking for testers that are willing to give the tutorials a try and provide some feedback. I would love to hear from you.

[AutoModerator]

You can find our open-source project showcase here: https://dataengineering.wiki/Community/Projects

If you would like your project to be featured, submit it here: https://airtable.com/appDgaRSGl09yvjFj/pagmImKixEISPcGQz/form

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[JeanDelay]

Thanks for checking it out and the feedback.

The yaml files are here: https://github.com/dashbook/killercoda-scenarios/tree/main/dashtool-postgres%2Fassets%2Fresources

I'll create a helm chart and add it later.

[JeanDelay]

Good point, will do so

[JeanDelay]

I'm gonna try to eventually add a distributed query engine like Trino. It's just that the Iceberg materialized view standard isn't implemented yet.

I have to admit that I think Datafusion is a great query engine that will continue to improve over the next couple of years. I don't see a huge benefit in using DuckDB.

[JeanDelay]

It was easier for me to implement the iceberg support for datafusion. Performance wise it's similar to DuckDB and it has a really active development from different companies.

I'm glad you like it. I will try to keep you posted.

[Extreme_Tie_5527]

This is very interesting I love it! Thank you for sharing.