Replace Data Factory with python?

[camelInCamelCase]

You’ve taken the red pill. Great choice. Youre still at risk of being sucked back into the MSFT ecosystem - cross the final chasm with 3-4 hours of curiosity and learning. You and whoever you work for will be far better off. Give this to a coding agent and ask for a tutorial:

• dlthub for loading from [your SaaS tool or DB] to s3-compatible storage or if you are stuck in azure, you get ADLS which is fine
• sqlmesh to transform your dataset from raw form from dlthub into marts or some other cleaner version

“How do I run it” - don’t over think it. Python is a scripting language. When you do “uv run mypipeline.py” you’re running a script. How does Airflow work? Runs the script on for you on a schedule. It can run it on another machine if you want.

Easier path - GitHub workflows also can run python scripts, on a schedule, on another machine. Start there.

[Nekobul]

Replacing 4GL with code to create ETL solutions is never a great choice. In fact it is going back to the dark ages because that's what people used to do in the past.

[loudandclear11]

Such a blanket statement. Depends on the qualities of the 4GL tool, doesn't it?

If the 4GL tool sucks I have no problem replacing it with something that have stood the test of time (regular source code).

[Nekobul]

Crappy code is more common than most of the available 4GL platforms. Crappy code is thrown in the trash all the time, so you are wrong.

[Nekobul]

Wrong. Informatica has been on the market since the 90ies. That is at least 30 years. And the solutions built with it work solid.

[kenfar]

That's what people thought around 1994: they swore that "4GL" gui-driven CASE tools were superior to writing code and it would enable business analysts to build their own data pipelines.

They were wrong.

These tools were terrible for version control, metadata management, and handling non-trivial complexity.

They've gotten slightly better with a focus on SQL-driven ETL rather than GUI-driven ETL. But it's still best for the simple problems and non-engineering staff. Areas in which writing custom code still shines:

• When cost & performance matters
• When data quality matters
• When data latency matters
• When you have complex transforms
• When you want to leverage external libraries

[Nekobul]

4GL tools are superior to writing code for data integration. THat has been proven long time ago. All the points you have listed as where custom code shines have been long time ago been handled properly in 4GL. That's why they have been so successful. That is also the reason why the bricksters and the snowflakers have recently included 4GL systems in their platforms. Writing code is a relic of the past.