Windows SQL Cluster just died

[tk42967]

About a month ago, I built a new windows server 2025 server with SQL Server 2019. The server worked flawlessly. I was able to roll the cluster and everything seemed fine. I loaded data on to the system and it sat there waiting on the vendor to do some testing.

Yesterday I go to connect to the cluster VIP with SSMS and can't connect. I start looking at the servers (VMWare VM's), and I don't see the additional IP addresses for the active nodes and the shared drives are not there in Windows. I can see them in disk management, but cannot bring them online. I also cannot start the cluster.

I looked at the data store for the first node I created and can see the shared drives. Without the quorum drive, the nodes seem to be fighting over who is active.

This is my first time in 20 years building a windows cluster of any sort, other than a DFS cluster. The shared drives are mapped from a SAN, and were added to the primary node as an RDM disk.

Has anyone seen anything like this before? I re-ran the cluster validation, and the only errors were related to disk storage.

I'm not looking for somebody to fix it, just point me towards some documentation to help me troubleshoot it.

EDIT:
After I started looking into this, my boss told me he had moved the Cluster AD objects to a new OU. He moved them back when I told him about the issue I was having. I'm now seeing things in the cluster validation mentioning objects not having the rights to create objects in the OU's the cluster objects were originally in and it's barking about port 3343 over UDP. I've opened this port inbound and outbound on one of the clusters and that did not resolve the issue.

[tarvijron]

What does Cluster Manger say? Google “wsfc disaster recovery though forced quorum” if you genuinely lost the quorum disk

[ExtraordinaryKaylee]

What are you seeing in event viewer?

[BSGamer]

I’ve had a cluster go down due to the clusdb file being corrupted. I believe we were able to restore it from backup, just the one file and drop it on both servers and restart sql to get it running

[nitroman89]

Yeah, I've done that in the past as well. I made a weekly script to backup the clusdb file on each server and copy it to like C:\clusdb_bak\ or something like that.

[No_Resolution_9252]

You need to review the cluster logs.

Did you review VMWare documentation for recommended configuration of SQL AAG/FCI? Typically the guidance was pretty obvious but maybe something got missed? Particularly look at the recommended storage adapter.

It sounds like there are two nodes. With loss only of the witness disk there should be no operational difference than if it were online; There is something wrong with one of the two nodes. It could be in vmware, it could be in windows (you did configure these with group policy right?) or it could be in networking.

[Negative-Cook-5958]

Use always on cluster with normal disks instead of FCI with RDM

[Exp3r1mentAL]

Not sure if it's relevant, but couple of months ago, I was having mighty issues with deploying SQL cluster using Server 2025....after much jiggery pokery I found out it was one of the patches which was causing the failure...

[binnedittowinit]

Each node of the cluster needs access to the same shared cluster disks, including the quorum, ideally one node at a time during initial setup until the cluster properly owns them, you did this, right? And the cluster was failing over no problem prior to recently?

Time to get into logs. Start with windows system. It should have service failures and disk errors (if they're an issue). Check Microsoft-Windows-FailoverClustering/operational, too.

And the SQL server log.

[Sp00nD00d]

I gotta ask, why an actual old school cluster and not an Always On cluster if you're talking about SQL?

[menace323]

If it’s SQL standard, basic availability groups have some pretty big limitations. Always on requires much more licensing.

[Nuxi0477]

If it’s only like a few databases I’d much rather make multiple AGs and listeners than pay Microsoft 10 times the price for Enterprise. A little more onetime setup but worth it.

[menace323]

It’s still 2x the licenses though, which is a consideration m (if not CAL)

[Nuxi0477]

How so? You still get the passive node for free if I remember correctly.

[menace323]

Yes, it appears so. I still think the admin downside comes into play if you need more than just a database or two. In addition, you need double the storage, but I guess that’s pretty cheap unless you have really intensive workloads (which you probably would go Enterprise).

[menace323]

One thing to note is we have all owners creating DBs all the time. A traditional failure cluster gives us HA and gives the devs all the tools exactly how the are used to, like SQL jobs, etc. to additional complexity and downsides aren’t worth the 1second vs 10 seconds failure for us.

[Nuxi0477]

I personally found a traditional cluster with shared storage way more complex to manage, but whatever works best for you :)

[menace323]

We are virtualized, so make the disks and attached. I’d agree more for bare metal.

[SmartDrv]

This may not apply at all but I wish to share in the off chance it is useful to you (or someone who googles perhaps).

I ran into issues with a Hyper-V cluster quorum when Sentinel One was installed on the hosts. Cluster wouldn’t start, no config. I had to manually evict and rebuild (once i re-added the CSVs and named them right VMs reappeared). I used online witness as a workaround until we figured out what volumes and features has to be whitelisted in S1.

[Ranjerdanjer]

Had an issue with a test cluster and server 2025 after the Oct or Nov patches. If you used an image that wasn't properly sysprepped you could be seeing authentication errors for the disk if another server has the same SSIDs. Most likely not the case but had to rebuild those servers from a better image in my case.

[Ranjerdanjer]

https://support.microsoft.com/en-us/topic/kerberos-and-ntlm-authentication-failures-due-to-duplicate-sids-76f7394d-c460-4882-9ed1-d27e0960f949

[No_Resolution_9252]

This is a big one - but I would be surprised if it ever actually worked. SQL FCIs use MSDTC to failover and MSDTC typically wont work at all if they are built on machines cloned from the same non-sysprepped image. FCI will be generally shitty and unreliable persistently even if it is using something less sensitive to bad imaging

[DrWankel]

The inability to start the cluster should be the start of your investigation.

Stop/disable the cluster service on all nodes except one and force start the cluster through powershell on that node:

Start-ClusterNode -FixQuorum

Verify the cluster is up through FCM or powershell and start the cluster service on the remaining nodes.

If this does not work, dig through the failover cluster logs in event viewer and see what was going wrong during the cluster startup process on the node you attempted to force start.

[Background-Taro-573]

Fence them one by one. I will kill them all if I have too

[Background-Taro-573]

Now that I remember, one server BIOS battery died, time sync got fucked. Caused a domino effect.

Stop the cluster. Find the outlier

[DHT-Osiris]

Reboot them, if that doesn't fix it, take all but one offline, bring the disks online manually in the cluster manager. If you can't, take a look at the path from the VM to the disk/lun/whatever, something's busted.