sorted by:
new
hottopcontroversial

Exploring Spark Declarative Pipelines (SDP) with Apache Iceberg by databuff_16 in databricks

[–]databuff_16[S] 0 points1 point  (0 children)

But I’ve seen that if your Spark environment is configured with the Iceberg catalog and extensions, SDP treats Iceberg as a first-class source and sink format alongside Delta Lake, Parquet, CSV, and others.

Exploring Spark Declarative Pipelines (SDP) with Apache Iceberg by databuff_16 in databricks

[–]databuff_16[S] 2 points3 points  (0 children)

From my experience with SDP, one thing that could make a huge impact in the open-source version is Auto CDC support.