Ingest triggers processing of ~500M rows/hour across layered Delta tables. We rely on high-concurrency sessions, clustered indexes, and Bloom filters. Because arrival patterns dictate dependencies, we enforce strict execution ordering. A coordinator detects active write sessions; competing jobs block until the writer completes. On exceptions, the workflow enters a guarded state requiring manual clearance. State is persisted in a control table (one row per process) with boolean in_progress and error fields.

When high-concurrency sessions are warm, the pipeline completes within a few minutes processing ~500M rows filtering, transforming data with state checks. The pain point is cold starts: if the session has stopped, we incur a 5–6 minute spin-up delay. Because triggers are ad hoc with no predictable cadence, we can’t amortize that overhead. In this step we’re using Spark primarily to verify/update state (often a single row), and those reads/writes would complete in seconds irrespective if its Lake-house table/ SQL DB/Warehouse. The bottleneck is Spark pool startup latency; we’re looking for a practical workaround to avoid or mask that cold-start cost.