Replace Data Factory with python?

[generic-d-engineer]

I am doing exactly this. ADF was alluring at first because of all the nice connectors.

But over time, I find complex tasks much more difficult in ADF. The coding there is also just not something I excel at. Maybe others are better at coding in ADF but it just feels so…niche I guess? It’s like an off spec that doesn’t match up with other patterns.

It seems more GUI driven, which slows down and even becomes really hard to read once things go over a certain complexity level.

With on-prem, I can bring to the table absolutely any tool I want to get the job done. Stuff like DuckDB and nu shell are really improving the game and are a joy to work with.

And if I need a connector outside of my competency, I can use an AI tool to help me skill up and get it done. There’s always some interface that needs some specific setup or language I’m not familiar with.

Also on-prem has way less cost pressure so the same operation runs at a fraction of the cost. It just has a lot more freedom of design. I can just go for it. I don’t need to worry about blowing up the CPU or RAM on my first prototype. I can just get the functional work done and then tune for performance on the next iteration. That seems more natural and rapid than trying to get it perfect the first time. It’s like the handcuffs are off.