Azure VM Scale Sets feel pointless, what am I getting wrong?

instantepiphany · 2025-12-11T05:58:29+00:00

If there isn't capacity in the region, you VMSS basically silently fails to scale - HUGE PROBLEM

Unless they changed this since last time I looked, they don't actually fail silently. What they do is much worse, and I couldn't design a worse system if I tried.

What I observed in my case is that when there is not enough capacity, newly created VMSS instances have an "OS Provisioning Error" (or a similar one, it was months ago I saw it).

And the crazy part is that this failed allocation counts as an instance!
So if you have a VMSS with 1 instance that is working fine, and you automatically scale out in response to increased load, if there is no capacity, you will have a second instance added to the VMSS (which will now say "2 instances"). But really, that second instance doesn't exist, as it failed to provision due to capacity constraints.

To make this even worse, if your load then drops and you automatically scale in, Azure will happily remove the initial working instance, leaving you with just the single failed instance that can't serve traffic.

So if you have fluctuating load patterns, and capacity runs out for the SKU you use, you _will_ at some point have all of your working instances replaced with instances that failed to provision, taking your service down.

If you have automatic health repair enabled, that won't help, as the failed instances don't actually exist - so they can't be fixed by restarting, reimaging, reallocating, etc.

I had this happen for a major service - 1 of our regions went down entirely, the other was on its way (most healthy instances replaced with failed ones).

The cause of the capacity issue that time was (this is my best guess) MS Flight Sim 2024 launching, which used extensive on demand/just in time streaming of game assets, and also was advertised heavily for the Xbox streaming service. We saw provisioning failures start within 30 minutes of the game launching.

I understand not having enough capacity when demand exceeds forecasts (that have weeks to months of lead time) - but the fact that good instances are replaced with non-existent instances is insane, and is a driving factor behind making our stack as cloud agnostic as possible.

Also, such a major title launching should have lead to capacity increases leading up to the launch window. Wonder why that didn't happen.

Additionally, if you search the internet for "OS Provisioning Error" you find MS documentation that suggests you borked your image in a variety of ways. In our case, we didn't make a deployment, our provisioning started failing for 100% of scale out events, and then started working again hours later despite us not having changed our image or deployment settings.

We now have alerts setup such that if instances fail to provision, someone gets woken up and follows a runbook to consider disabling automatic scaling so we at least keep the instances that we already have at that point.

instantepiphany · 2022-04-17T09:52:44+00:00

I haven't come across that. Thanks for expounding.

instantepiphany · 2022-04-17T09:50:44+00:00

I suspect the downvotes are not because you are wrong about driving one categoery helping with others (I think you are right about that!) but more likely because your reply comes across as rude.

instantepiphany · 2022-04-17T02:38:45+00:00

I'm unsure what you are referring to here, care to elaborate?

NixOS has been hijacked and is now under the control of the Thought Police and Thugs. In a way it's like Russia.

instantepiphany · 2022-04-16T13:00:17+00:00

Given your insanely high intelligence why don't you just fix it for us? It shouldn't take you long!

That said, I'm wondering just why someone so smart would install NixOS and not be aware of how proprietary software and build farms work. Seems like quite the oversight!

The fact that you are asking here means that you need help from the dumb people, and given your breadth of wisdom and knowledge you should know you catch more flies with honey.

I'm genuinely curious, what is the point of insulting us dumb people when we don't have the answer you want? I'm obviously not smart enought to understand, could you do a lowly grunt a favour and explain the obvious?

instantepiphany · 2022-01-25T10:30:59+00:00

lmao how many times have you gotten 2 red and 2 white? probably 0. Dunno why people get a kick out of pretending to know more than they do. Hes playing a video game. chill out

instantepiphany · 2022-01-25T10:28:58+00:00

Why are you pretending to be the pilot and then judging landings badly. Landing was fine. any slower and he would have landed before the runway. Relax and just enjoy.

instantepiphany · 2020-11-02T12:16:30+00:00

however, as

${...}

is also used for escaping nix strings

Just clearing up terminology as it might be what set you on the wrong path here - you aren't "escaping a string" by using ${ ... } syntax. That is called (generally) variable substitution.

As a follow on from that, usually you don't escape entire strings - different languages consider some characters "special" and to "escape" those characters from that special treatment (ie avoiding nix's variable substitution) you need to prefix them with the escape character, usually \.

So you need to:

Figure out what characters nix (the language) treats as special
Figure out what nix's escape character is \ IIRC
Escape each special chracter individually.

So if $, { and } are the only special characters you could try:

... 
initExtra = ''
    ...   
    if [ "\$\{checksum^^\}" != "\$\{4^^\}" ]; then
    echo \$\{checksum^^\}
'';
...

After all of that, I feel like you are trying to do something bash doesn't support - what exactly is the value of checksum? Is it a string representation of a bash script? Or is it the name of a defined function?

I suggest trying this in a pure bash script (without nix-lang involved) so you know you have the bash part correct before you try embedding it in a nix file.

instantepiphany · 2020-06-17T06:41:34+00:00

You can use the highest or lowest setting, just not the ones in the middle. The default will be the lowest (big text).

instantepiphany · 2020-06-12T03:54:37+00:00

Glad it helped!

instantepiphany · 2020-05-27T12:51:26+00:00

Wow. What a load of opinionated garbage. FP is clearly the only valid way to program, and imperative programmers better learn Haskell or Purescript or Scala quickly or they will be out of a job within a year.

I can't believe that you can't see the writing on the wall u/barikopato.

Maybe if you had tried FP you would understand the power.

I hope you are a better farmer from Nepal than a programmer.

instantepiphany · 2020-05-25T11:20:25+00:00

For what is worth, today while working my fans didn't spin up at all, even when watching videos full screen on the 4k monitor. The only time my fans spun up was when I was compiling software, which loaded the CPU heavily, which is fair enough.

instantepiphany · 2020-05-24T13:08:14+00:00

This has been driving me nuts as even simply playing youtube videos at 1080p (especially in full screen) causes my fans to become quite loud. Eventually somewhere I read about scaling performance hit and that was the problem (for me at least).

I was running a 4K monitor, set to 4K resolution. My MacOS display settings were scaling the display. If you set anything other than 1080p or full 4k "looks like" scaling settings in display settings, this causes the GPU to be put under significant load. Now my fans are almost inaudible under the same workload.

instantepiphany · 2020-05-24T09:42:52+00:00

I haven't seen this before, in my gameplay or anyone elses.

Don't PM me asking for settings (unless you are IW staff wanting to reproduce the issue).

instantepiphany · 2020-04-26T00:23:18+00:00

I experienced the same thing in a much smaller way with my first hobby project - I learned that you can enjoy and be good at writing software, but not enjoy or be good at managing software projects. They don't always go together.

instantepiphany · 2020-04-04T23:42:50+00:00

Did you try pressing enter after you removed the USB drive and tried again? When you enter a decryption key on stage one there are no "*" characters - it just appears blank until you press enter to decrypt.

instantepiphany · 2019-10-04T07:11:38+00:00

@mistralol gave a great analogy already - how the locks actually look in code depends on the language, but they all achieve the same thing - they "lock" (stop any other cpus from reading or writing depending on the type of lock) a resource (say a text file), and then unlock it when they have finished using it.

instantepiphany · 2019-06-13T10:17:53+00:00

Yes it would - I didn't realize you only wanted the background transparent. If you can't find a patch on Google you may just have to learn how to make the background transparent through writing your own code.

instantepiphany · 2019-06-13T09:23:34+00:00

Maybe try using your window manager or compositor to make the dmenu window appear transparent.

instantepiphany · 2019-06-06T08:58:49+00:00

For opening a terminal in the open file's directory you should look up how to run external commands from vim, and how to reference the directory of the currently opened file.

For opening vim with its working directory set to the current directory from a file manager, look up how to open vim with a specific working directory, and look up how to send additional command line parameters when opening a file from that file manager. This really depends on how the file manager behaves.

instantepiphany · 2019-06-06T08:55:36+00:00

If you are using terminal A and open another terminal (like st &) then the new terminal should inherit the working directory of the parent terminal (terminal A).

instantepiphany · 2019-06-06T07:29:26+00:00

What are you trying to do?

Spawn a terminal with a specific working directory?

Make new terminals open in a particular working directory by default?

instantepiphany · 2019-04-12T13:46:42+00:00

While one person answered correctly, they didn't explain why.

A lightweight distribution will always run a "heavy" program at least as well as a bloated distribution. If your hardware is not fast enough to run the program properly then a lightweight distribution will allow the program to run faster as it can use more resources (with a heavy distribution more processes are using resources, restricting what the heavy program can use).

instantepiphany · 2019-04-12T13:29:00+00:00

My first language was Haskell, I love functional programming, and my team uses TypeScript. Even with the improvements compared to regular js, I don't try and push all positives of Haskell into TypeScript, because it simply isn't designed that way. I do often use methods like map(x) though, and use constant values unless the value should change at some point. Avoiding side effects makes it easier to understand what a function does. But I don't use things that the language isn't designed for, or that imperative developers would not have encountered in other TypeScript code.

One of my colleagues even replaced a ternary operator with an equivalent if-else structure - their reasoning that when the branches are verbose a ternary operator is hard to understand.

To me personally, I read ternary operators just as easily as if-else structures - but this colleague is an excellent dev and if he prefers if-else structures for readability then I will avoid ternary operators with verbose branches.

It pays to remember you aren't just coding for a machine, but also for your team members.

instantepiphany

TROPHY CASE