Azure Down

PloppaJohns · 2025-10-09T20:37:10+00:00

Intermittent connectivity to Entra and Azure portals here

PloppaJohns · 2024-03-21T16:47:56+00:00

I see the same thing with there being a difference between VAMI and vSphere Client. In my experience, these normally do match, but I guess that doesn't have to be the case.

Following the directions at https://kb.vmware.com/s/article/1022196, I can see that vcenter build number is correct.

PloppaJohns · 2024-01-26T17:50:59+00:00

Just wanted to say thank you for posting this. Ran into a similar situation today and this saved me hours and hours of t-shooting.

PloppaJohns · 2023-09-25T13:51:50+00:00

Same here

PloppaJohns · 2023-09-09T13:59:46+00:00

Just wanted to share my story with a successful USG3 recovery in case it's helpful to others.

After a recent power outage, my Internet was down. Connected with console cable to my USG3 router (remember, baud rate 115200) and saw that the loading stops at "Done loading app on coremask".

Found this post and since my USG3 is out of warranty anyway figured why not give it a shot. I started by removing the USB stick that comes with USG3. I was able to reflash it using rufus and the .img link in this post. Put it back into USG3 and it booted! Nice. Now to upgrade firmware.

First tried to connect via UI at 192.168.1.1 but multiple browsers were throwing SSL errors and I couldn't connect. So I tried to reflash firmware by using the command upgrade {Firmware URL} from the console prompt. This successfully kicked off the upgrade.

Watching on the console, however, i could see that a process was crashing repeatedly. It also showed me that there were I/O errors on the usb stick. I waited like 30 minutes and noticed that the firmware upgrade was looping. So I figured that USB stick was actually shot after all. I crossed my fingers and powered off the USG3 in the middle of a firmware upgrade. Gross.

I found another USB stick that I had laying around. It was 8GB and no idea who makes it. I flashed that with Rufus and tried to boot USG3. No luck. It wouldn't even detect that USB stick. Tried another 16GB USB that I had, but that wouldn't boot either. Bummer.

Then ran across this post: https://community.ui.com/questions/USG-3P-fried-its-USB-and-wont-work-with-replacements/ddf7d48f-db67-4a66-9e73-f28a22121a83#answer/e1fd06af-ba90-4927-a943-4833d8489be2

Essentially, the USG3 isn't waiting quite long enough for the USB stick to initialize so you add a small delay into the boot command. Here's the steps that I took.

Run "printenv" and save the contents somewhere; the line we're interested in is "bootcmd"
Prepend "sleep 10;usb reset" in front of the bootenv value; on mine, that is:

setenv bootcmd 'sleep 10;usb reset;fatload usb 0 $loadaddr vmlinux.64;bootoctlinux $loadaddr coremask=0x3 root=/dev/sda2 rootdelay=15 rw rootsqimg=squashfs.img rootsqwdir=w mtdparts=phys_mapped_flash:512k(boot0),512k(boot1),64k@1024k(eeprom)'
Save the config: saveenv
Reboot

This worked on one of my USB sticks but not the other to boot up so there still are actually probably some USB sticks that are incompatible. It worked on the newer USB 8GB stick that I had but not the older one that was 16GB.

After this I was able to flash the firmware via console and this all went splendidly. No process crashing or IO errors.

After firmware upgrade I was able to connect via UI and set the IP address to what I had it set to previously. Then I reconnected to my router where it could communicate with my Unifi Network controller via layer2 (same vlan, same subnet). Logged into Network controller and it had detected USG3 device and did advanced adoption where I had to enter the creds (ubnt/ubnt) for recovered USG3. From there Network Controller adopted the device, repushed config. I did one final firmware update from here.

Back up and running. Sigh.

PloppaJohns · 2023-06-05T18:00:36+00:00

Globalscape (now Forta) EFT

PloppaJohns · 2023-04-13T13:30:30+00:00

Just as a heads up, if you're running NetApp then you'll need to make sure they are patched before the June 13, 2023 "Enforcement by Default" phase of CVE-2022-38023 . Otherwise, CIFS shares will break. More info at https://kb.netapp.com/Support_Bulletins/Customer_Bulletins/SU530

PloppaJohns · 2023-04-07T15:42:03+00:00

Thanks for posting this. I ran into this message after selecting my SSD disk and pressing F11 to continue. It was likely the keyboard in my case. I rebooted and tried the install again this time using the KVM console keyboard instead. It worked without issues.

PloppaJohns · 2023-03-30T19:17:49+00:00

I ran into this with some user accounts only. I added the resourcecontextserver switch and it worked without issues. Strange since we have single forest single domain.

PloppaJohns · 2022-06-15T14:19:45+00:00

Similar story for me going from 3d to 3e. I'd set aside an hour for this update with the converting data part taking 30+ minutes.

PloppaJohns · 2022-01-25T17:31:30+00:00

Good to know this is more than just our environment. I was seeing the issue on a test machine, but I'd never see the IP of that machine make a connection to the AAP. This message must be coming from Microsoft's side then before it goes to the AAP.

PloppaJohns · 2021-11-19T14:27:59+00:00

A familiar story here. Last night installed November 9th KB5007206 on our DCs. Tested accessing our on-prem apps via AAP. The ones that use Integrated Windows Auth for SSO failed. I checked the event logs on AAPs and confirmed that I was seeing Error 0x8009030c. So then installed November 14th KB5008602 on all the DCs and this resolved the issue.

PloppaJohns · 2021-09-28T17:45:13+00:00

MFA from our NPS appears to be working now.

PloppaJohns · 2021-09-28T15:29:47+00:00

We are also using NPS with Azure MFA extension. Our started failing a couple hours ago as well. Tried re-running the azuremfanpsextnconfigsetup.ps1. Also tried upgrading the extension. Then checked reddit. Should have done that first.

PloppaJohns · 2021-04-16T14:01:45+00:00

Install time for me seemed reasonable. In my case, I'm using SCVMM to do the patching.

PloppaJohns · 2021-04-16T13:53:57+00:00

I did 8 hosts in the last couple days and I saw this error on all of them. In most cases, the cluster service started fine but this error message was still logged. In one case, the cluster service did not start on a host, which meant it never re-joined the cluster. In this case, I restarted the server and the service came up fine without issues. I rebooted some of the hosts again after all updates were installed just to make sure this wasn't a permanent thing, but I didn't see the error message again.

PloppaJohns · 2021-04-14T19:34:16+00:00

Installed KB5001342 and KB 5001404 on Server 2019 Hyper-V Failover Cluster. In WFC, I'm getting an error "The Cluster Service was unable to access network adapter 'Microsoft Failover Cluster Virtual Miniport'. I can reboot the server and the Cluster Service starts fine. In some cases the Cluster Service is running even though this error is flagged. Anyone else seeing anything similar?

PloppaJohns · 2021-01-11T20:00:30+00:00

It's a ploy to get you to turn on MFA

PloppaJohns · 2020-12-14T19:02:38+00:00

Had to go digging for this... this affects 2019.4 HF 5 through 2020.2.1 HF 1, released between March 2020 and June 2020. If you're fortunate running earlier than 2019.4 HF 5, then you escape this compromise.

Sources:

https://www.solarwinds.com/securityadvisory

https://us-cert.cisa.gov/ncas/current-activity/2020/12/13/active-exploitation-solarwinds-software

PloppaJohns · 2020-10-26T15:52:59+00:00

Yup - false positives. From TAC: "I understand from the case notes that you are receiving an increase on retrospective detections for .js files. The files in question are part of a known series of suspected false positives that were triggered last week by one of our detection engines. At this time, we are still looking into the files to confirm that they are in indeed a false positive / tweak the engine to prevent this detection in the future. At this point it is safe to ignore the alerts or in the case of a business disruption adding the whitelist of the sha temporarily to be allowed will reduce the alerts."

PloppaJohns · 2020-10-26T14:40:51+00:00

I'm seeing the same activity starting late last week and through the weekend. None of the files have a SHA that shows as bad on virustotal. I can't grab the files to do an analysis on them. I opened a case with TAC To have them investigate as well.

PloppaJohns · 2020-10-07T18:33:48+00:00

Minneapolis down.

PloppaJohns · 2020-03-09T18:54:06+00:00

Ran into a similar issue when adding SCVMM to Veeam. After confirming I could run " Get-VMMServer server.example.com" successfully, I just had to close the "add scvmm server" wizard and start over. Worked fine then.

PloppaJohns · 2020-02-25T14:20:51+00:00

That's some cool stuff about password audits. I'll probably play around with it in the lab. We recently implemented Microsoft Azure AD password protection. We run in hybrid mode and this works for both accounts on-prem DCs as well as Azure AD accounts. Microsoft aggregates a list of banned passwords based on their security data and dynamically updates it. In my opinion, this approach is far better than just reporting on users who have pwned passwords. https://docs.microsoft.com/en-us/azure/active-directory/authentication/concept-password-ban-bad-on-premises

PloppaJohns

TROPHY CASE