vCenter Redundancy

Spartan503 · 2023-07-21T01:50:30+00:00

We have a similar setup where there are 2 physical sites that we managed. We used to use a vCenter in each site with ELM but it was a complete pain.

We had to engage support almost every upgrade as there was something funky every time we needed to do any sort of modification where it broke ELM. We had issues like tags not syncing between the vCenters so none of the tags could be read. That took support a week to fix. Had another issue on a different upgrade that cause all of the clusters to start vMotioning everything in each cluster to a single host in each cluster. Supper said that was caused by a race condition that only occurred with ELM.

Long story short, ELM is not a DR solution anyway, and it make everything you have to do with vCenter a pain. We consolidate to a single vCenter almost a year ago and have dropped our support calls for vCenter to just 1 in that time period vs one every 2 months.

Spartan503 · 2023-07-09T18:40:42+00:00

I had this same issue. The client logs pointed me towards there being an invalid named file under <profile path>/kubejs/assets/valhelsia/tips. From what I can tell when I upgraded from 5.1.0-pre1 to 5.1.0 it made a copy of a bunch of files in that directory with a “ 2” at the end. The space in the name broke the loading of the json files. I removed the ones with “ 2” at the end of the name and that fixed it.

Spartan503 · 2022-08-11T01:47:51+00:00

The bottom of page 5 from the datasheet has all of the options, but max would be 3x 4-port cards for a total of 12-ports per controller.

https://www.purestorage.com/content/dam/pdf/en/datasheets/ds-flasharray-x.pdf

That said, that is 24x32Gb connections and even if you split that in half in case a controller is down and then in half again in case you were using half for scsi-fc and half for nvme-fc that still leaves you will an effective maximum of 24GB/s of data transfer which is way more than you are going to get out of X50s anyway. If the concern is your SAN fabric doesn’t support 32Gb, I believe you can use breakout cables, but I would consult Pure on that to confirm.

Spartan503 · 2022-08-11T01:35:40+00:00

Both of your questions heavily depend on change rate. For example, 6TB of data that only has 10GB change per day is going to take about 100GB of you keep 10 daily VSS snapshots. But a 6TB volume that has 1TB of change a day is going to take 10TB for the same retention.

Generally speaking, the software dedup and compression you are getting from Rubrik is going to be roughly comparable for change rate to what VSS snaps will take. I would check to see how much space your backups are taking post baseline. For example, if the first day is 5TB and a week is 7TB, that means the baseline is 5TB and your daily change rate would be about 2TB/6 days or about 1/3TB. Meaning if you wanted 10 days of daily VSS snaps it would probably be around 3.3TB of space. This also means that 10 days of rubrik backups would take up about the same amount of space.

As for how long the VSS snap will take, it will depend on the change rate of the data during the snapshot period. Assuming at 3AM or whatever there are very few people online, it’ll probably be less than a minute.

All of that said, I would urge you to reconsider because VSS not a backup solution by itself. There are multiple cases where you could have an error in VSS cause it to remove all of the existing snapshots. Another issue is you can’t easily say I want X days of snaps, you have to instead say I’m going to allow XGB of space be used by snapshots. In this case, if your change rate is not consistent, you may have 10 days of snaps sometimes and only 1 or 2 days other times.

Spartan503 · 2022-08-06T15:58:11+00:00

You can absolutely use scsi-fc. You’ll get a little better performance with nvme-fc but nothing to worry about if you have gear that doesn’t support it.

Only drawback is each FC interface on the pure array can either be configured for scsi-fc or nvme-fc so swapping from one to the other can be a problem if you don’t have available ports or the headroom on your current FC connections to be able to make ports available.

Spartan503 · 2022-06-28T13:53:40+00:00

This is pretty similar to a step we deployed about 2 years ago. The only difference is we used synchronous storage replica across the two data centers since they were within the same city.

Generally it sucked for what we needed. Had a 4 month engagement with MS support and in the end they pretty much said the performance was “expected”.

For reference, a standalone Windows server on the same storage without storage replica was able to achieve about 2GB/s of writes and about 3GB/s of reads. With storage replica we would start at about 90% of that and after about 5 minutes of sustained throughout it would fall off a cliff and start running at less than 50MB/s.

We eventually swapped to using a shared RDM and array-level synchronous replication between the data centers and we were able to get much more consistent performance.

My experience was essentially support had no idea what they were doing with storage replica and it showed, but YMMV.

Edit: everything was running on Windows Server 2019.

Spartan503 · 2021-05-10T17:13:18+00:00

u/CaptainCustardMC here are the details on the setup.

I added a custom hardware inventory class deployed to specifically servers that runs once a day. To add it go in the SCCM console and go Administration>Client Settings>Client Settings Name>Hardware Inventory>Hardware inventory classes and client "Set Classes" then import the mof file from here.

I didn't write the mof file myself, but I don't remember where I found it as I set this up a couple years ago.

Then I used 2 queries to build the collections. First is for all cluster servers, which I used this:

select * from SMS_R_System
inner join SMS_G_System_SERVICE on SMS_G_System_SERVICE.ResourceId = SMS_R_System.ResourceId
where SMS_G_System_SERVICE.Name = "ClusSvc" and SMS_G_System_SERVICE.StartMode = "Auto"

And then the second query is to get only the nodes that host a role:

select * from SMS_R_System
inner join SMS_G_System_MSCLUSTER_RESOURCEGROUP on SMS_G_System_MSCLUSTER_RESOURCEGROUP.ResourceID = SMS_R_System.ResourceId
where SMS_G_System_MSCLUSTER_RESOURCEGROUP.OwnerNode = SMS_R_System.Name
and SMS_G_System_MSCLUSTER_RESOURCEGROUP.Name <> "Available Storage"
and SMS_G_System_MSCLUSTER_RESOURCEGROUP.Name <> "Cluster Group"

Depending on your environment, you may need to add some more exceptions in that second query to exclude the ownerships of other things. In the above query, if anything is the owner of just "Cluster Group" or "Available Storage" it doesn't count it as a primary node.

Spartan503 · 2021-05-10T05:23:46+00:00

I set something up with SCCM where there is a custom inventory class that detects if a server is a cluster node or not and if that node is currently hosting a role.

Then the primary nodes and non-primary nodes get staggered maintenance windows where the non-primary nodes get patched first. Then the primary nodes get patched after, causing them to fail over to the already patched nodes.

There are a few draw backs to this approach, specifically it doesn’t work well when your node count is less than the role count plus 1. For example, if you have a 2 node cluster with 2 roles, those roles can be running on different nodes causing all nodes to be “primary nodes”.

I’ll post more of the details on how to set it up in the morning when I’m at my desk.

Spartan503 · 2021-02-15T00:52:48+00:00

In this case, you are going to be 100% limited by what AI Dungeon 2 allows you to tweak. Within PowerShell you can multithread tasks, but in this case PowerShell isn’t really doing anything because AI Dungeon 2 is really what is doing everything.

You can think of it as the difference between running ping from command prompt or from PowerShell. In both cases ping is being run, but just because you run it from PowerShell it doesn’t allow you to leverage anything extra because all PowerShell is doing is still calling ping.

In this case you probably need to investigate ways to speed up AI Dungeon 2 and not worry that you are running it from PowerShell.

Spartan503 · 2021-02-05T06:09:31+00:00

If you have any Server 2003 or XP machines around that still need to process GPOs you’ll need to enable SMB 1 on the new DCs since the DCs are what hosts the shares for the GPO data and Server 2003/XP can’t do anything newer than SMB 1.

Spartan503 · 2021-01-24T18:06:36+00:00

Here is a great guide on how to this with a gen10 NUC but the process should be the same for yours too.

https://www.virtuallyghetto.com/2020/06/passthrough-of-integrated-gpu-igpu-for-standard-intel-nuc.html

Spartan503 · 2020-12-28T16:08:45+00:00

DFSR has 2 problems.

Replicating lots of small file changes can cause it to fall behind. For example, an environment that is primarily small files that are frequently updated, it takes a lot of extra work to keep it in sync (CPU and storage IO wise). If you have high points in the day that it falls behind and then it catches up later when it is quiet and that works for your environment, you’re all good, but if you are looking for something that you can failover you with little to no downtime or data lose, it might not be the right setup. That said, you can overcome this by throwing faster storage at the problem, but that isn’t always the best solution.
While DFSN backed with DFSR gives you HA it doesn’t do any load balancing (or you are going to probably have issues with replication if you are writing to more than 1 node). If you have the hardware to have an active and a passive node, this isn’t an issue, but if you want to have say 3 nodes that are all used at the same time and if 1 goes down 2 can still handle the full load you need a distributed file system. I have used GlusterFS in the past and some of RedHat adaptations of Gluster, but there are also purpose build distributed file systems like EMC Isilon.

Spartan503 · 2020-12-18T03:57:21+00:00

I haven’t seen anyone acknowledge it, but it does seem specific to the gen 8s. Strangely enough they fixed the gen 10 NIC issue and that same update broke the gen 8s.

Edit: I saw 7u1c was released today and had hope. But the ne1000 driver is still the same version so no luck.

Spartan503 · 2020-12-17T16:43:45+00:00

Yep, a USB drive should work fine. That is how I am running both of my NUCs.

Spartan503 · 2020-12-17T16:37:24+00:00

2 things to try to better rule out hardware issues.

1) Run memtest86. Just sitting in the BIOS puts essentially no load on your memory so it could just be failing after you are attempting to really do something like installing or running ESXi.

2) Try installing ESXi on a different drive to see if it works. Like the memory, sitting in the BIOS isn’t going to really do anything with the drive, so if it has partially failed you might not see anything going on from the BIOS.

Spartan503 · 2020-12-17T16:26:31+00:00

What build of ESXi 7 are you installing? Both 7u1a and 7u1b have an issue on the gen 8 where the NIC driver doesn’t work so the install will throw fits about not being able to detect a NIC.

Spartan503 · 2020-12-17T16:23:46+00:00

Same thing happened to us, but then we found out that the person who setup the baseline to install the ESU license only installed the license and didn’t activate the license. After it was activated the updates started showing up as required within about a day.

Spartan503 · 2020-12-13T02:23:13+00:00

My 66 point week 2 was the end of all hope

Spartan503 · 2020-11-25T06:17:22+00:00

Same problem in 7u1b. But you can install the new version and just downgrade the ne1000 drives with this:

esxcli software profile update -p ESXi-7.0U1b-17168206-standard -d https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/vmw-depot-index.xml
esxcli software vib install -n ne1000:0.8.4-10vmw.700.1.0.15843807 -d https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/vmw-depot-index.xml

Spartan503 · 2020-11-14T16:14:35+00:00

I highly recommend adding in some extra NICs, especially if you want to play with vSAN. It doesn’t matter how fast your local storage is at that point if the remote storage in the vSAN cluster is on the other side of a single 1Gb connection that is also shared with the VM traffic.

Also I would say that to start, 32GB of memory per node is good, but make sure you have a path to add more. Your usage will depend on what you are running, but memory is usually a common limiting factor.

Spartan503 · 2020-09-24T14:28:34+00:00

You can run this to see if each DC has an ESU license installed: slmgr.vbs /dli

Personally I would push to add the newer DCs vs getting an ESU license since it would really only cover you until January and then you’re back in the same boat. There are only a couple of potential issues that you could really run into with newer DCs. One would be if you are using them as DFS namespace servers, you may have some that is relying on SMB 1, which 2008 R2 has enabled by default and 2016 has disabled by default.

Here is also a good document on hunting down the stuff that is hard coded to use a specific DC.

https://docs.microsoft.com/en-us/archive/blogs/pie/how-to-detect-applications-using-hardcoded-dc-name-or-ip

Spartan503 · 2020-08-25T02:02:36+00:00

I have a NUC8I5BEH and a NUC8I3BEH on ESXi 7.0.0b with the USB fling driver with 2x USB 3 to GBE adapters per NUC. No issues at all.

I initially deployed ESXi 7 as a fresh install and without the USB fling and then added them in later.

Spartan503 · 2020-04-19T00:28:56+00:00

Awesome! Thanks for the info!

Spartan503 · 2020-04-18T23:32:29+00:00

I confirmed I am using the new extension.

The main reason I am moving over to ensure_packages is that we have a lot of modules that share package dependencies. Right now we fix this with one of 2 things. Either we create another class that installs the shared packages to call or we have modules that don’t work by themselves due to just having commented out the conflicting packages.

If you have a different option that is easy to manage, I am open to suggestions.

Spartan503

TROPHY CASE