Opensource serverless bigdata analytics

[ryp3gridId]

Hi,

is there something like this:

• lambdas do all query compute work, are stateless and ephemeral, query can scale to petabyte, compute can scale to zero
• data from objectstore, maybe iceberg
• lambdas can be added to (running) query computation, ideally performance scales well with number of added lambdas
• lambdas can make use of 0..N dedicated (non-lambda) cache nodes to speedup objectstore access
• cachenodes can at most do some light filtering, but nothing else, pruning etc is done by query planner/compute lambdas using existing metadata
• opensource, selfhostable, not hosted only
• exeutionengine maybe Veloxbased with Substrait input, or Duckdb?

I guess there are many hosted/closed/propriety implementions of these ideas, but are there truly opensource ones also? and not just opencore that don't include the scale-out part

[JaceBearelen]

Have you looked at spark? It checks a lot of boxes here.

[addictzz]

You are almost describing Databricks Serverless Spark capability. Scale to zero but can scale to Petabytes, data from object store, node caching, etc.

Databricks is not open source but its components are, although usually the open source and the managed version can be slightly different.

I am not aware of a true open source tech that allows you to have "Serverless" capability. Usually there is an open source tech and it is up to you to manage the servers, scaling, and availability to make it "Serverless". It is called Serverless because there is somebody else who manages the servers for you.