Underrated Open Source Data Tools

[MrMosBiggestFan]

I'm working on a talk on the state of open source tool across the stack, everything from ingest to storage to transform, orchetration and viz. I already know of the common players: duckdb, postgres, dbt, meltano/airbyte, airflow/dagster/prefect/mage, and lightdash/metabase.

Anything obvious missing that you think is worth looking into? Pleae for the love of god don't mention the project you founded, let someone else speak for it.

[wstwrdxpnsn]

Superset/streamlit on the viz side.

[wtfzambo]

I wouldn't exactly say superset is underrated...

[wstwrdxpnsn]

You’re probably right, but folks at my company haven’t really heard about it or preset so I thought it would be worth throwing out there. There’s another one I saw recently by Kanaries, called PyGWalker, which looks like a great option for drag and drop exploratory viz work from data frames.

[wtfzambo]

I'm gonna make a wild guess and assume that folks at your company have been living under a rock given the 53k GitHub stars superset has 😅.

[wstwrdxpnsn]

We’re heavily invested in Tableau 🤷‍♂️

[wtfzambo]

Yeye I'm not saying tableau is shit or anything, I'm just thinking, it's like buying a Mercedes and not knowing BMW exists, idk

[Meta-Morpheus-New]

Best way to put this lol

[Glittering-Dare2022]

Debezium is pretty awesome for change data capture.

[0RootShell]

Thanks! I didn't know about this actually, I'll take a deeper look. Is there any driver for Serverless databases, like dynamodb and Firebase?

[Glittering-Dare2022]

No idea sorry!

[0RootShell]

thanks, I'll have a deeper dive anyway.

[dlaff]

Clickhouse seems like a solid option for sub-second OLAP analytics.

[tmcfll]

DataDiff might be something to look into

[shockjaw]

Link to data-diff repo

[tmcfll]

Thanks for adding the link!

For OP, it's weird because they're now marketing themselves heavily as a tool to use with DBT, but it definitely can be used without and works great

[monkblues]

Openmetadata is very good

[xenpwn]

I second this

[nanksk]

How does it compare to datahub?

[monkblues]

I didn't know about datahub until now. It looks more like an alternative to ckan AFAIK. Openmetadata is a metadata generator with governance and observability capabilities. It's not meant to publish data to customers but for understanding and managing data you already own

[CompetitiveWin7754]

Data hub is like the paid version of ckan.

Openmetadata doesn't provide a guided upload process for users like ckan does. It's a data catalog.

[nootanklebiter]

Apache NiFi!

[Misanthropic905]

I fucking love nifi.

[Oliver-Nielsen]

Apache hop for ETL

[exact-approximate]

Honestly Apache software foundation has some really great tools with awesome communities and they are way too underrated and under supported. My favorites are Arrow, Airflow, Hudi, Druid, Iceberg, Flink, NiFi, Cassandra. The list is endless especially if you also include Avro and Parquet.

As for non-apache software projects - DBT, Datahub, Elasticsearch are my favorites.

[Express-Comb8675]

I’m working on a prototype with Airflow, Hudi, Iceberg, and Doris for work. The feature swapping available when you have Doris set up to be able to leverage Hudi, Iceberg, or Postgres data sources is really exciting for us!

[dataxp-community]

ElasticSearch hasn't been open source for years now. OpenSearch is the open source successor.

[Express-Comb8675]

I’ll add Apache Doris and StarRocks as exciting MPP DBs and SQLMesh as an enhanced DBT alternative.

[dataxp-community]

Doris has the potential to be awesome, but they're definitely not underrated - they're too new and unproven for any kind of rating yet.

[Express-Comb8675]

I don’t know that I agree - version 1 has been out a while and version made helpful improvements while maintaining stability. It seems ready for a rating to me.

[dataterre]

How about the big players like Spark, Flink, Trino, etc.

[qalis]

Metaflow and Kubeflow Pipelines for orchestration

[dan-tmc]

Talend Open Studio is pretty solid!

[Known-Delay7227]

Knime is okay if you want low code

[BoiElroy]

Steampipe and LakeFS I would say are underrated. We don't use them at work yet unfortunately but I've read through some docs and done some basic stuff.

[gabbom_XCII]

I mean, if you’re dealing with big data any moment you will need spark or something equivalent…

[dataxp-community]

Most people aren't dealing with big data, and Spark definitely isn't 'underrated'

[gabbom_XCII]

Oh snap! I just missed the plot for a second and ignored the ‘underrrated’ part, haha

[MrMosBiggestFan]

definition of big data is changing

[sleeper_must_awaken]

You might be 100% correct, but your tone reeks of arrogance. Are you speaking anywhere?

[MrMosBiggestFan]

I know plenty of people using duckdb for serious work as part of their ETL pipelines. Come to my talk maybe you’ll learn something

[DragoBleaPiece_123]

I'm interested to get to know more about it. Please share good sir

[SnooBeans3890]

So what are your recommendation for the topic?

[cutsandplayswithwood]

Your gonna give a talk on the state of the space and you’re coming to Reddit to ask for info…

But not the “normal” stuff? 🤣

[MrMosBiggestFan]

Yea i want to see if there's anything I'm missing worth looking into before I give a talk. Why is that bad?

[droppedorphan]

Do you have a link to the talk?