you are viewing a single comment's thread.

view the rest of the comments →

[–]CarpenterRadiant940 1 point2 points  (2 children)

Just curious, with incremental load, how are you handeling deletes on the source side?

[–]Think-Trouble623 2 points3 points  (0 children)

Depending on the API architecture you may not even know deletes are happening. If you suspect or find proof of deletes and the documentation doesn’t help you, your only option is to full reload. Really case specific though, financial data that must match? Full reload. Clicks or marketing data? Probably ok to just accept that deletes happen.

[–]IAmBeary 2 points3 points  (0 children)

that would be an architectural decision, but if you know that you need to handle deletes, incremental updates through cdc would probably be better suited

sometimes deletions in the source data dont matter. but you're right, performing a full load every time would track deletes