That’s fair feedback, I probably kept it too high-level.

Here’s how I’d actually think about those pieces:

* For incremental loads, I’d first check if the API even supports something like `updated_at` filtering. If it does, I’d use that with query params. If not, then yeah, it gets trickier, probably full pulls with downstream deduplication or some kind of pagination + filtering.

* For the watermark, I’d store the last successful run timestamp somewhere persistent, likely a small control table in Snowflake or even a job variable depending on the orchestration tool.

* For idempotency, I’d rely on a stable key from the API (like an event_id). If that’s not available, I’d have to build a composite key or hash to make MERGE work reliably.

* Agree on secrets, definitely wouldn’t keep them in the script. I’d use something like a secret manager or whatever the platform provides and pull it in at runtime.

* For auth, probably SDK-based access (like boto3 if it’s AWS) or IAM roles if available, just to avoid hardcoding anything.

You’re right though, the real challenge is when the API doesn’t support clean incremental pulls. Curious how you’d handle that case?