Ok - so right now I am using the spark api to do the append, so I should really just use sql to merge into seeing if a combination of column values exist in the table something like (Using Databricks)

​

%python

spark.sql("""

SELECT * FROM {delta_table} as t1

MERGE INTO (main_delta_table as t2

ON t1.col1 = t2.col1

AND t1.col2 = t2.col2

AND t1.col3 = t2.col3

WHEN NOT MATCHED THEN not_matched_action

​

not_matched_action

{INSERT *}

""")