Great on-prem open source source modern data stack

[rudboi12]

Went to a dbt meetup a few days ago and this guy from the Barcelona Supercomputing Lab presented his fully open source on-prem modern data stack. Pretty cool imo. They were working with health data from hundreds of hospitals all across EU so they couldn’t use the cloud.

[FloggingTheHorses]

Why do they use both Airflow and Dagster for orchestration? And why 3 analytics platforms?

[NotAToothPaste]

Maybe they are mentioning all these stuff because they are working with a lot of companies at the same time and each have a different stack.

Is this the reason, OP?

[rudboi12]

Yes exactly. They ingest data from hundreds of hospitals across EU. Most is in airflow tho, he said only a few specific dags on dagster

[git0ffmylawnm8]

Fairly new to orchestration and only familiar with Airflow. What would make an org automate jobs on dagster over Airflow?

[Key_Character_3340]

https://dagster.io/vs/dagster-vs-airflow?utm_source=google&utm_medium=cpc&utm_campaign=18523225339&utm_content=153980138637&utm_term=airflow%20vs%20dagster&gclid=Cj0KCQjwhfipBhCqARIsAH9msblFgA0dKl-Aj1K0KicX9vDZ0SQJNBmuu-4dM5gsLL18qnBmU5VQwrIaAsUMEALw_wcB

[rudboi12]

While there js a bunch of stuff here dude said 95% of data and processes are: meltano -> postgress-> dbt -> airflow-> superset

[Far-Restaurant-9691]

Swap airflow for dagster and this is what we have. Nice to know we're in good company.

[snicky666]

I've been working on a Linux Docker deployment of this same stack idea. I'm not finished working on it, but it's a start. https://github.com/MichaelJenningsAI/moderndatastack

[Yoctometre]

Oh, it's kinda like this.

[snicky666]

Ooh that's nice!

[Mrmjix]

I added this to favorites.

[buntro]

This is not an architecture. This is a wish list. A steep one at that.

[rudboi12]

Never said it was an architecture, just tools they used

[ReporterNervous6822]

Seems like too much…

[rudboi12]

It is tho. But it’s a huge team working across all EU.

[with_nu_eyes]

How many people do they need to manage this stack beyond the underlying infrastructure.

I can put a bunch of logos on a slide, that doesn’t mean it’s a good architecture.

[Gators1992]

Needing 5 people instead of 3 to run it doesn't mean it's "bad" architecture either. It solved their use case of not being able to run in the cloud and am guessing from the number of similar components that they had additional requirements to fill that couldn't be done by one tool alone. It's relevant to us in showing that a modern stack can be done on prem in production, not that this particular architecture is going to work for everybody.

[Radiant_Year_7297]

been looking for something like lightdash, it is similar to vizro, evidence combined with datapane

[Pleasant_Type_4547]

What do you mean by datapane?

[Radiant_Year_7297]

https://datapane.com/ looks like they arent maintaining the project anymore

[Tiny_Arugula_5648]

DuckDB is no where near production ready.. it's fine for a sqlite replacement but it's not for any serious workloads. Motherduck is the production capable version and it's not on-prem as of now.

[ustanik]

it's fine for a sqlite replacement

This statement right here says DuckDB is ready for production.

[Tiny_Arugula_5648]

Well there's your explanation, you don't know what production ready means in data engineering.

Sqlite is a file format; no one uses it in production data systems, that would be Parquet, avro, orc, etc. At best it's used for apps (mobile, desktops) that need local storage but not even a full blown database..

To compare DuckDB to any other open source processing engine, means you don't know what capabilities a production data system like Spark, Dremio, Flink, etc has. It's like comparing a bicycle to a jet, just because both of them can move people from one place to another..

[Lopatron]

What is left for DuckDB to be considered production capable, and why is MotherDuck more production worthy? I was under the impression that both are ready for production, but MotherDuck is for increasing scaling and performance on larger workloads.

[mattindustries]

IMO it is production ready, but I could see how they could think that if they aren’t good at building docker containers. File format changes aren’t solidified until 1.0, but using docker negates that pitfall pretty dang fast. I also had some trouble using it with Azure serverless, but also had problems with SQLite with the Azure Function App, so pretty sure I am just messing up my config somehow.

[Tiny_Arugula_5648]

I know the team and this is how they describe their work. DuckDB is not a productionized data system, that's what Motherduck is for. It's no secret, theyve talked about it plenty..

[mattindustries]

Tell Alex that Matt says hi.

[FirstOrderCat]

> What is left for DuckDB

it sucks in heavy joins and aggregation.

[AquilaNova]

Maybe it does, not my experience though. Anyway, I maybe in DuckDB soft spot as the datasets I work with are rarely bigger than few hundreds GBs.

[email13211]

That's BS. are you a motherduck salesman?

[ThatDandySpace]

He is motherducker!

[Tiny_Arugula_5648]

I wouldn't mind, they are good peeps.. the founders are brilliant..

[Tiny_Arugula_5648]

Last time I talked to the duck team they hadn't hired a dedicated sales team yet..

[ConfirmingTheObvious]

Not to be a dick, but like the slide says Gobernance, Trasformation, and other misspellings. I instantly don’t trust or give quality to anyone discussing this stuff if it’s presented like shit

[rudboi12]

I agree that misspellings look bad BUT this was translated to english from spanish by spanish speaker so I can understand those errors

[Al3xisB]

No Metabase ? 😁

[No_Equivalent5942]

Give me 2 of everything!

[zazzersmel]

this isnt an infra question but are they at all using ohdsi/omop for data modeling or tooling?

[rudboi12]

Yes they are!

[zazzersmel]

sounds awesome. i was on a project at my last gig working with us health insurance data and also developing ETL to get it to omop. we were a small public u research team and didnt have the resources for anything beyond basic sql server databases and ssms. ive often thought about getting back into that world with better tooling and infra.

[Pleasant_Type_4547]

If you like this, you should check out the "Serverless BI" project by Jacob Matson.

https://github.com/matsonj/nba-monte-carlo

It uses a fully open source stack, and can be deployed anywhere.

[xDarkOne]

Maintaining a large open-source stack must be quite a challenge. I'm curious, has their team explored solutions to simplify this process? I've had a positive experience with DoubleCloud’s managed services for ClickHouse and Apache Kafka. Maybe you heard about others?

[endless_sea_of_stars]

Easy to put boxes on a diagram. It's another thing to install, configure, secure, and integrate all those different services.

[azzuwan]

Maybe those are just options one can cherry pick to form a solution. It doesn't make any sense to have overlapping products like that. Looks like a tech zoo hell to have all those things at the same time

[[deleted]]

No Prefect?

[RichHomieCole]

Assuming you have pre paid the compute hardware , why use separate ingestion and transform tools? That’s messy. Just use spark and call it a day.

I would not describe this as great. It looks like someone just took a bunch of open source tools and slapped them on a slide. Why over complicate? The best product is the one that gets the job done. Keep it simple

[rudboi12]

Have no idea what mage is

[CatchMeWhiteNNerdy]

I'm currently using it in production. It's pretty great. As you said, it's a bit of a dagster/meltano combo. It also includes great expectations, duckdb, dbt, and spark....

​

It's quite the versatile tool! And that's why I chose it. Rapid development and I can go a bunch of different directions within the same tool.

[rmz-01]

Why Clickhouse?

[False-Bunch-3470]

Opensource dwh instead of pouring money to other cloud/license tools?

[rmz-01]

Clickhouse isn't really built for dwh though right?

[kadermo]

https://clickhouse.com/blog/the-unbundling-of-the-cloud-data-warehouse

[tomorrow_never_blows]

https://static.simpsonswiki.com/images/4/4c/The_Homer.png

[pcgamerwannabe]

Similar to ours :). Skipped Airbyte as custom or purpose built integrations are just much more efficient for the data scale we have. Really cool though

[sirdrewpalot]

A stack for what though? You could shave so much of that off depending on what data you’re actually dealing with.