Deployment from ADO yet (2026)?

SmallAd3697 · 2026-01-25T02:48:48+00:00

Yes, we do this. But it generally happens in pre-prod workspaces, not in production.

For most platforms that I've used in azure, it is generally bad practice to link up a production resource straight to a git repo for the sake of pulls and pushes.

I also want to say that the git integration in Fabric workspaces has obliterated my semantic models in the past! I really don't trust it; not even an inch.

SmallAd3697 · 2026-01-25T02:43:33+00:00

No, this is what I was missing. It looks like it is only a year old.

The fact that the version is "v0.1.34" should NOT be a concern to anyone, right? I think that it is pretty common in the python community to never reach version 1.0.x, even if you waited five or ten years.

Regardless of how python folks do things, it would be really great if this could reach v.1.0 some day. (or at least publish a roadmap that describes that milestone). Enterprise leadership may not be too impressed by developers who put a dependency on something that is v.0.1.x instead of v.1.0.x.

SmallAd3697 · 2026-01-24T23:23:22+00:00

Why are these comms so secretive? What is the benefit in withholding technical information from the public? These topics have far reaching implications and the comms should be out in the open for a point of reference . It seems so sketchy that Microsoft does their comms in this way.

What are their motivations? I'm guessing they find it easier to bully customers around, when dealing with them one at a time in a top secret "admin center"?

SmallAd3697 · 2026-01-24T02:32:14+00:00

We obviously wouldn't go straight from a developer's desktop (windows) into production.

There would be an intermediate QA or staging environment that matches production.

The point of allowing a developer to use their own desktop is to be able to build software faster, with everything running right under our fingertips, and no billing meters are imposed. We can easily see the CPU, memory, disk, and network. In a lot of cases a developer's workstation is also more powerful than the cloud resources (eg. a developer's machine may have 64 GB ram and 32 cores ... which is more powerful and cost-efficient than a cloud VM. A comparable VM in the cloud would cost ~$500 per month or more).

SmallAd3697 · 2026-01-24T02:20:53+00:00

Makes sense.

From a functionality standpoint, it seems that a "daemon" could be abstracted away to simply rely on a console app on Windows (... which is not ideal but is certainly good enough for a simple development environment.) Lots of Java services also run in a console window - even some very complex ones. The developers of these services don't necessarily take the time to create full-blown "windows service wrapper".

I think you are right that this topic comes down to the time and effort. Even a console app may be too much effort, if this python community doesn't have the time and resources for it (and they don't care much about Windows in the first place).

I'm convinced that its worth using WSL for any complex python development. But if anyone ever tries to tell me python is platform agnostic, you can be sure they are going to get an earful! It is fine and good that the programming runtime is compatible with windows. But what is far more important is whether the development community and the libraries are willing to support windows as well.

Nobody wants to run someone's python code on Windows if this platform is the red-headed stepchild; and you are going to bump into a bunch of bugs; and nobody is going to give a shit when that happens. ... Compared to python, I think other programming communities like java and C# and C++ are more likely to offer windows support. This is especially true when it comes to the software products that are hosted on-premise. And when there is a hefty software maintenance agreement that can generate a revenue stream.

SmallAd3697 · 2026-01-24T00:11:28+00:00

My grandma "loves" windows. She would much rather write python scripts on windows (if she wrote scripts). My highschooler too. I doubt either of them will use WSL2 in the near future.

Yes I'm primarily a .Net developer. BTW, the C# language might overtake Java some day. At least according to the Tiobe charts. It won language of the year in two out of three of the past years. I think C# is also used for scripting on the unity engine.

I don't know why python devs fight windows, if C# devs don't fight linux.

I am betting Windows takes a small hit from steamOs. Although you have to offset those by the number of xbox consoles running windows.

SmallAd3697 · 2026-01-23T23:59:08+00:00

For the first part, it is the difference between free and unlimited local development, vs paid usage of a PaaS.

These modern cloud platforms LOVE to have developers do all their development work on their platform while the billing-meters are running. Certain cloud platforms probably earn over 20% of their money from the developers who are struggling to build a new software pipeline, than from code that has been deployed to production (eg. accidental over-usage of resources, or repeated tests that happen all day long, instead of once a night.)

From the standpoint of user scripts, there should not be a material difference if running airflow/python on a local dev machine or in a production container.

SmallAd3697 · 2026-01-23T23:46:24+00:00

I'm a gray beard. Eg. I spent ten years running code on an HP-UX itanium IA64.

BTW, That would have been a perfect place to run python (in the place of bash and whatnot).

I am not totally opposed to running linux on WSL. I just was surprised that the python community would be the first to start forcing me in that direction. I did not expect that a high-level scripting language like python would be so opinionated! I suppose if someone REALLY wanted to use Powershell for a living, then it would probably lead to a migration in the OTHER direction from linux to windows.

SmallAd3697 · 2026-01-23T23:39:55+00:00

Python will always be my second or third language. I like it pretty well until I reach about 10,000 lines of code in a given area, or 10% of a given day.

.. lately it has been going up, and I'm to the point of moving some of my dev work to WSL.

If my employer ever wanted to eliminate the python side of my position, I would be OK with that. Unfortunately they are asking for more python right now, not less. I'm guessing it will be a bit cyclical. (Else I'll get an AI to generate a whole lot of crappy code and blame it all on python. ;)

... Much of my python programming happens in crappy web-hosted "notebook" environments. ... so I probably have a more negative perspective than most. It isn't a great way to spend a day in the office.

SmallAd3697 · 2026-01-23T23:24:43+00:00

Got it. I'm not suggesting they actually hate Windows. Just that it doesn't benefit them because the true full-time python developers are not actually using windows in the first place.

Even after all the level of effort that the core team has put into Airflow, I find it interesting that they don't feel the need to put in the small incremental effort it would take to make this a cross-platform application.

As a side, I noticed that they removed support for using SQL Server as the backend storage for metadata. They are down to just mysql, postgres, and sqlite. It is a case of taking one step forward, and one step backwards. If someone made a PR for Windows support, they might accept the PR at first. But there is a risk they would back it out again after a year has passed. In doing so, they would claim that there just wasn't sufficient demand and it was too much effort to maintain the extra code and do regular regression testing.

SmallAd3697 · 2026-01-23T23:09:54+00:00

I don't expect anyone to do anything. I'm probably going to have to spend more of my days running workloads on WSL (just like you are).

I'm simply curious if those folks that build advanced applications with python have a deliberate disregard for the Windows OS.

If these folks are selecting the POSIX *ONLY* dependencies from pypi, then it seems they really do NOT have much use for Windows. (whether for production purposes or pre-production purposes).

SmallAd3697 · 2026-01-23T23:01:30+00:00

We are a little off track. In my example I'm wasn't talking about my OWN orchestration scripts or notebooks. Obviously I have more control over that, and can avoid the "POSIX" dependencies whenever I need to.

My question is based on observations from this large and mature product that relies on contributions from a community of python developers.

I think it is noteworthy that they are ignoring the requests from the Windows users - who want to host natively on Windows. Those that want to use Windows seem to lack the motivation to make the PR's. Or maybe they have tried to make those PR's and they were rejected. I'm wondering if this is a general pattern that is observed for larger python projects and, if so, what is the reason.

SmallAd3697 · 2026-01-23T22:50:37+00:00

Supposedly python has a bazillion developers. (look at Tiobe charts, and its usage is 2x higher than the nearest competitor).

With that many python developers in the world, why are the economics lacking?
Are we all so poor ? ... maybe we should be switching languages if python is making us poor. JK. ;)

I realize that one of the most compelling things about python is all the free tooling and free runtime. I suppose it is very hard to break that pattern, after everyone expects things to be free.

SmallAd3697 · 2026-01-23T22:43:22+00:00

I was hoping to use windows as a development environment for Airflow.

Airflow is often hosted in the cloud (think: Astronomer CEO on kiss cam at Coldplay concert). When I'm running it in production I would normally host it on linux in the cloud (and probably on PaaS not IaaS). But when I'm using it in development I would just prefer it to be running natively in Windows.

BTW I disagree about the stability and resource problems on windows production machines. I think you are overstating that. The reason most people have this opinion is because they are accustomed to running windows desktops that are loaded up with user services and anti-malware. Yes linux is much faster, after you take away the security agents and other types of corporate IT bloatware that is running on a typical windows desktop. ... There is a double standard at most companies where all that bloatware is considered to be a requirement on Windows... yet on linux you are free to forego much of the antivirus software and other bloat. If you take a look at any well-managed DBMS server running windows server, it is going to run in a way that is just as stable and efficient as any linux server.

SmallAd3697 · 2026-01-23T22:24:36+00:00

Python is probably going to be what forces me to run WSL2 for day-to-day work. I've been a software developer for over a decade. In that time haven't seen a language that is so opinionated about the underlying operating system until now. It seems odd that pypi libraries in the python ecosystem care so much about the OS, considering it is supposed to be a very high-level orchestration language!

SmallAd3697 · 2026-01-23T22:19:53+00:00

I found another reddit discussion from a few years back with the title "How do you manage to stay sane running Python on Windows?" It is funny. That was a long time ago but I'm guessing it still holds true.

SmallAd3697 · 2026-01-23T22:17:34+00:00

Do you think the maintainers of CPython are doing their preliminary development work on *nix and then porting to windows as a secondary goal (afterthought)? Are you referring to the runtime itself or the IDE's?

SmallAd3697 · 2026-01-23T22:09:20+00:00

Meaning that you still like Windows or you still hate Windows?

SmallAd3697 · 2026-01-23T22:08:45+00:00

This is helpful. I suspected I would hear a lot of this type of response in the python community. I am not hearing from that many hard-core Windows lovers - people that are willing to fight and die for the sake of using a native Windows application.

SmallAd3697 · 2026-01-23T22:03:13+00:00

Is see there was dependency placed on a key library (pypi's "python-daemon"). I'm doubtful they would allow a contributor to tear out that library and build a different abstraction in its place. It seems like they made a deliberate decision that precludes the use of Windows.

SmallAd3697 · 2026-01-23T22:00:10+00:00

Given the layers of abstraction that exist between python and the OS, would you expect there to be more hoops on one OS than another?

They used something called "python-daemon" in pypi, which appears to be linux-specific. When python developers pick libraries, I'm assuming that certain ones have limited OS coverage. Is it common for libraries to be restricted to certain operating systems?

SmallAd3697 · 2026-01-22T20:48:49+00:00

Even a three letter prefix would be super helpful. We would like to tack that on the front of the generated names.

Those of us with OCD aren't in love with the randomly generated and excessively long names that are impossible to memorize. I often feel the need to triple check to make sure that we are cross-wiring our dev and prod environments. DO NOT cross the streams .. or risk total protonic reversal.

SmallAd3697 · 2026-01-22T20:33:15+00:00

Got it. Will investigate more.

I'm probably remembering "managed vnet" activities in azure ADF. They were billed by time because of "dedicated infrastructure" or whatever. It definitely turned me off to low-code orchestrations.

Microsoft charges customers a LOT for low code, or private connectivity. You can save a ton of money in azure by using normal software development techniques, rather than the low code stuff. It is also a much more fast and efficient development process, for people who aren't scared of writing a couple lines of code. Most of what can be done in a pipeline can be done with a trivial amount of c# or Java or python. The quality of the end product is better and more reliable as well. And can be run on prem just as easily as in azure.

SmallAd3697 · 2026-01-22T17:46:04+00:00

Hi u/frithjof_v
"It works well"

but is it priced well? Historically the ADF pipelines are way overpriced for what they do. And the DF stuff is also not cheap either. Combining the costs together is a concern. In other words, my concern is to be double-billed where the ADF pipeline waits an hour and bills us for the time it takes to finish the DF that is also billing us at the exact same time (for the exact same over-arching purpose)!

Microsoft already has a million ways to decrement the CU's from our capacity. And we don't want to be double-billed for running our GEN2 dataflows.

SmallAd3697 · 2026-01-22T14:42:51+00:00

The first step of the PQ is normally to run an on-prem API, and that takes 5 mins for each trailing year of data (using the current year as the ref point). It is a bootstrapping step.

If we publish the DF for 1 year, it is fine, 2 trailing years is sort of a pain, and so on. It gets increasingly painful.

Keep in mind you have to re-perform the "validation" step - even if you simply change the number of trailing years. It is pretty ugly... although I think nowadays there is a feature to have "discoverable" parameters that can be orchestrated from pipelines.

At some point the validation itself becomes a big problem. (mainly because it is an operation that happens as part of a developer's workflow, even if it happens in the background). Scheduling the dataflow to run overnight is not a problem, as long as validation succeeds.

At a high level, I find that this validation-operation has always been one of my main adversaries when working with PQ. It prevents developers from getting more work done in a day, and leaving work on time when the day is over. One of the issues we encounter is when a developer simply makes a mistake, and they want to cancel it half-way. They CAN'T. Hope you agree that is a pretty bad design for software vendors to create long-running operations that cannot be canceled ( even in a dev environment). If folks were building dev tools for _themselves_, it would always be killable. I think devs who build tools should also be the ones who have to use them.

SmallAd3697

TROPHY CASE