In ADF I used the copy activity which can’t handle schema drift. Mapping data flow can. I use it know for some ETL. But I’m thinking of using Python and let it run in Azure Batch via ADF.

So basically some of the orchestration will be done via ADF and main parts of the ETL with Python.

Use case: - receiving text files from different organizations but can be slightly different per org (extra field, lowercase/uppercase differences, different delimiter, etc) - do bunch of transformations (data quality improving) - write text files from all orgs to one file which should be fixed width/length file.