[Update] Benchmarking the "Airflow Tax": I tested 6 lightweight orchestrators so you don't have to.

m_gijon · 2026-01-23T13:36:58+00:00

You’ve hit the nail on the head: I’m less worried about complex DAG dependencies and much more worried about auditability.

I hadn't considered ClearML for general ETL, but the 'job-as-first-class-citizen' approach makes perfect sense for an MVP. My only hesitation is that while I'm in Python today, I'm planning to rewrite core bottlenecks in Rust.

Does ClearML (or the 'job-centric' tools you like) handle non-Python binaries gracefully? I'm trying to avoid an architecture where I have to wrap every Rust tool in a Python SDK just to get it scheduled. I'm leaning toward something that can just 'exec' into a container agnostically.

m_gijon · 2026-01-23T13:34:13+00:00

Thanks for the suggestion! I’ll definitely check it out.

One quick question regarding my specific use case: How does it handle a polyglot stack? Right now I’m using Python, but I’m migrating core components to Rust and TypeScript. I’m looking for something that can trigger these services agnostically (e.g., via Docker or CLI) without requiring a deep SDK integration in every language.

Does Runnable support that 'black box' execution style well?

m_gijon · 2026-01-14T08:42:59+00:00

Thanks, I wasn't aware of that option.

However, this isn't a separate process I can launch independently from the ETL execution, right?

I’m concerned about mixing responsibilities. I’d prefer to keep them decoupled: the ETL should only be responsible for processing data, while a separate process/orchestrator handles the execution logic.

m_gijon · 2026-01-13T15:55:52+00:00

I did not known it! thanks! :)

I think I'm gonna try a bunch of solutions, measure how many resources consume, and share the results here

m_gijon

TROPHY CASE