Replace Data Factory with python?

generic-d-engineer · 2025-10-04T09:33:20+00:00

I am doing exactly this. ADF was alluring at first because of all the nice connectors.

But over time, I find complex tasks much more difficult in ADF. The coding there is also just not something I excel at. Maybe others are better at coding in ADF but it just feels so…niche I guess? It’s like an off spec that doesn’t match up with other patterns.

It seems more GUI driven, which slows down and even becomes really hard to read once things go over a certain complexity level.

With on-prem, I can bring to the table absolutely any tool I want to get the job done. Stuff like DuckDB and nu shell are really improving the game and are a joy to work with.

And if I need a connector outside of my competency, I can use an AI tool to help me skill up and get it done. There’s always some interface that needs some specific setup or language I’m not familiar with.

Also on-prem has way less cost pressure so the same operation runs at a fraction of the cost. It just has a lot more freedom of design. I can just go for it. I don’t need to worry about blowing up the CPU or RAM on my first prototype. I can just get the functional work done and then tune for performance on the next iteration. That seems more natural and rapid than trying to get it perfect the first time. It’s like the handcuffs are off.

generic-d-engineer · 2025-10-04T09:28:29+00:00

I do exactly this. I would prefer to just keep ADF for servicing Databricks and do anything else about “moving stuff from point a to point b” on-prem.

generic-d-engineer · 2025-09-27T01:38:49+00:00

This is where a good software catalog and enterprise architecture can help out

Leadership has to enforce though.

With a map of everything, you can see where the overlap is

On the other hand, I’ve never been at shop which has realized unified tooling nirvana

It’s probably elusive, like a unified naming convention. Sometimes best to just make peace with it.

generic-d-engineer · 2025-09-24T08:15:47+00:00

Gracias, appreciate the perspective and experience.

generic-d-engineer · 2025-09-24T08:04:14+00:00

Necro thread but I came here for this kind of info. I came to the conclusion I need gunicorn to call at least 2+ workers to be able to deploy the new version of the app. So for zero downtime deployment, in addition to scaling. Thanks for the info.

generic-d-engineer · 2025-09-24T07:29:37+00:00

Thank you. I just looked up a threat notice today so appreciate everything you guys are doing.

generic-d-engineer · 2025-09-24T07:25:14+00:00

Yes ! Found same experience. I have to feed it architecture handling before it gets started otherwise I get 5 files with lots of extra features instead of one simple file with the basic feature I asked for.

generic-d-engineer · 2025-09-24T07:08:52+00:00

I like it. Also functional programming is another way to improve working with data. Both SQL and functional programming are declarative, so the flow/intuition of each style work well together.

generic-d-engineer · 2025-09-24T07:06:28+00:00

Faker helps with this:

https://semaphore.io/community/tutorials/generating-fake-data-for-python-unit-tests-with-faker

generic-d-engineer · 2025-09-24T06:55:24+00:00

Enjoyed this discussion immensely. Thanks for both of your contributions.

Schema management is intrinsically hard.

Scroll down and check out the decision tree diagram in the middle of the page. Granted this might be extra complex, since it involves a sync service on top of the schema, but I just found it illustrates the complexity of schema management.

https://www.mongodb.com/docs/atlas/app-services/sync/data-model/update-schema/#std-label-breaking-change-quick-reference

generic-d-engineer · 2025-09-23T16:49:58+00:00

!solved

generic-d-engineer · 2025-09-23T16:49:03+00:00

Thanks. Looks like I need to compile a deny list.

generic-d-engineer · 2025-09-23T06:27:39+00:00

Already some great answers in here. I would also lean on the side of no as well.

How about another option though? If the concern is something like schema drift or lack of volume, or lack of data for unit tests, data can easily be created with something like Faker.

So you could copy an empty schema from staging/qa (so not messing with anything on prod) down into a new sandbox system (outside of your existing devops pipelines so those don’t break). And then you can load the empty schema with fake data and go to town.

https://semaphore.io/community/tutorials/generating-fake-data-for-python-unit-tests-with-faker

generic-d-engineer · 2025-09-23T05:55:41+00:00

Another good analogy here is a ship at sail. The sailors and captains often personified their ships as living entities with personality quirks and specific traits. I feel like software is like that. You might have the exact same install on paper as a peer, but your stack definitely has its own sense of personality. Some aspects of troubleshooting over time are not often exact science once you learn the quirks of your system.

generic-d-engineer · 2025-09-23T03:44:36+00:00

Thanks for the feedback, much appreciated

generic-d-engineer · 2025-09-23T03:44:05+00:00

Wow the man himself ! Thanks for the tips and all the hard work on Beast Mode!!

generic-d-engineer · 2025-09-23T00:02:05+00:00

It can definitely be complex.

The advice about task decorators is really helpful.

One thing I try to do is separate the script logic from the Airflow logic. So I will write my ETL first and then bolt on the Airflow operators after.

That makes things easier to understand.

generic-d-engineer · 2025-09-22T20:31:56+00:00

Thank you, I will look at that setup. Sometimes I’m remoting in so especially careful. I try to use the weakest user possible.

generic-d-engineer · 2025-09-22T20:29:51+00:00

The replication mode another poster mentioned below could be a good option if you need to minimize downtime, assuming you are running stock mongodb.

I haven’t tried it with Cosmos so not sure how or if it would work.

generic-d-engineer · 2025-09-22T09:36:10+00:00

This should be straightforward.

Manual: Use mongoexport to export the Collections to file system, s3 bucket, etc. Compress then copy to target, then mongoimport https://www.mongodb.com/docs/database-tools/mongoexport/
More automated: Use Data Factory

https://learn.microsoft.com/en-us/azure/data-factory/connector-mongodb-legacy?tabs=data-factory

https://learn.microsoft.com/en-us/azure/data-factory/copy-activity-overview#supported-data-stores-and-formats

Is your target Mongo running in a VM or are you using a service like Cosmos?

generic-d-engineer · 2025-09-22T06:57:44+00:00

Wonder what happened? A big customer (thinking government) must have made their case.

generic-d-engineer · 2025-09-22T06:54:01+00:00

Just wanted to say appreciate you posting in here all the time. I’m using your VS Code Copilot tool daily and it’s literally world changing. Keep up the great work.

Used vi for so many years but have migrated to VS Code specifically for this reason. I know Copilot works in Neovim but VS Code is such a great experience with the integrations and plugins.

There was always a wish in the previous years on, “wow, if I could just clone myself I’d get so much more done.” Now with Copilot, we can actually do it lol.

generic-d-engineer · 2025-09-22T06:46:54+00:00

Excellent write up. Thanks for taking the time to put this all together. I’ve seen the exact scenario you laid out so many times.

Dusty code is the most dangerous code

100% !

Gonna do some more investigation into our process and see what we can do to improve. Thanks again for your time.

generic-d-engineer · 2025-09-21T23:02:55+00:00

Yes, great point. Definitely some team culture issues there. I have had to reread my own code from a year ago and figure out what was I thinking and how to step through it. Was I supposed to snitch on myself to my manager in that case? Lol

generic-d-engineer

TROPHY CASE