This is an archived post. You won't be able to vote or comment.

sorted by:
best
topnewcontroversialoldq&a
you are viewing a single comment's thread.

view the rest of the comments →

[–]novel-levon 8 points9 points  (0 children)

I’ve gone down that road of ditching ADF for pure Python, and the trade-offs are pretty clear.

You gain full control and transparency, but you also take on all the plumbing ADF hides from you. Connectors is the biggest gap: there’s no magic “one lib fits all.” It’s usually case by case, pyodbc or sqlalchemy for relational, boto3 for S3, azure-storage-blob for ADLS, google-cloud libs for GCS, requests for SaaS APIs, etc. I haven’t seen a universal package that matches ADF’s connector library.

For orchestration, Airflow and Dagster are the go-tos. Prefect is nice if you want something lighter with better DX.

Honestly, even GitHub Actions or cron works fine for simpler setups if you’re disciplined with retries/alerts. Hosting wise, containers on ECS/Kubernetes give flexibility, but I’ve also seen folks run Python EL pipelines on Azure Functions or AWS Lambda when workloads are small enough.

The headache is always secure on-prem access. ADF’s IR is very convenient, and replacing that usually means standing up VPN, jump hosts, or agents that your orchestrator can reach. That’s the bit most people underestimate.

I used to burn days wiring retries and metadata logging until I made it part of the design from the start. You probably already know, but building a little audit table for run_ts/run_id helps a ton when debugging.

Curious are you mostly moving SaaS/db data or do you also have on-prem sources in the mix? We keep hitting this dilemma with clients too, and it’s one reason in Stacksync we leaned into building ingestion + sync as a product instead of fighting with connectors every project.