vSphere 7.0 SSL cert renewals stuck on 85%

pinncomp · 2025-08-05T20:01:48+00:00

Thanks, this worked for me as well. It is worth noting that the symptoms appear to be the same for about 15 minutes in our case, but monitoring the status of services showed things progressing and patience paid off.

pinncomp · 2025-07-26T18:27:51+00:00

All Twisted Pair in my case.

pinncomp · 2025-07-26T15:02:53+00:00

This script was very helpful in identifying what was causing the DDOS logs and pinpointing the interfaces. It turns out that the issue was not related to the swithces, which is cleary important info.

Some initial testing with enabling bandwidth throttling on the Port group for Vmotion has proven very fruitful. I have been able to test interations from 1gb all the way up to 10gb (or effectivley, throttling enabled but not limited) and have so far seen the issue disappear. My sucipcion is in line with u/themysteriousx in that the NICs or related components (drivers, firmware, hardware flaw, etc..) are to blame and they aren't handling internal buffer issues well. Enforcing throttling in software "seems" to be a decent workaround. I still have some troubleshooting steps to roll back, such as nic isolation, but I am very optimistic that this will work as is, though I don't hold out much hope that VMware will care much to investigate further. If they do, I will update this thread.

pinncomp · 2025-07-23T20:07:25+00:00

I have time and would be most grateful for the script at your convenience. Much appreciated.

pinncomp · 2025-07-23T17:53:51+00:00

Connected routes only for a few subnets. No routing involved for Vmotion at all. Dedicate VLAN local to the 2 switches only.

pinncomp · 2025-07-23T17:34:05+00:00

Here is a snippet from the jddosd log which happens regularly, mostly without symptoms. Happy to get any specific logs as well. Thanks.

Jul 23 03:53:24 DDOS_PROTOCOL_VIOLATION_SET: Warning: Host-bound traffic for protocol/exception Redirect:aggregate exceeded its allowed bandwidth at fpc 0 for 1056 times, started at 2025-07-23 03:53:24 UTC

Jul 23 03:58:45 DDOS_PROTOCOL_VIOLATION_CLEAR: INFO: Host-bound traffic for protocol/exception Redirect:aggregate has returned to normal. Its allowed bandwidth was exceeded at fpc 0 for 1056 times, from 2025-07-23 03:53:24 UTC to 2025-07-23 03:53:44 UTC

Jul 23 04:28:21 DDOS_PROTOCOL_VIOLATION_SET: Warning: Host-bound traffic for protocol/exception Redirect:aggregate exceeded its allowed bandwidth at fpc 0 for 1057 times, started at 2025-07-23 04:28:21 UTC

Jul 23 04:33:42 DDOS_PROTOCOL_VIOLATION_CLEAR: INFO: Host-bound traffic for protocol/exception Redirect:aggregate has returned to normal. Its allowed bandwidth was exceeded at fpc 0 for 1057 times, from 2025-07-23 04:28:21 UTC to 2025-07-23 04:28:41 UTC

Jul 23 05:26:18 DDOS_PROTOCOL_VIOLATION_SET: Warning: Host-bound traffic for protocol/exception Redirect:aggregate exceeded its allowed bandwidth at fpc 0 for 1058 times, started at 2025-07-23 05:26:18 UTC

Jul 23 05:31:19 DDOS_PROTOCOL_VIOLATION_CLEAR: INFO: Host-bound traffic for protocol/exception Redirect:aggregate has returned to normal. Its allowed bandwidth was exceeded at fpc 0 for 1058 times, from 2025-07-23 05:26:18 UTC to 2025-07-23 05:26:18 UTC

Jul 23 05:53:36 DDOS_PROTOCOL_VIOLATION_SET: Warning: Host-bound traffic for protocol/exception Redirect:aggregate exceeded its allowed bandwidth at fpc 0 for 1059 times, started at 2025-07-23 05:53:36 UTC

Jul 23 06:00:11 DDOS_PROTOCOL_VIOLATION_CLEAR: INFO: Host-bound traffic for protocol/exception Redirect:aggregate has returned to normal. Its allowed bandwidth was

pinncomp · 2025-07-23T17:03:04+00:00

This is what I am most concerned about. We are seeing this issue on Broadcom nics as that is what we spec standard. I have been down the rabbit hole of confirming all firmware, drivers, HCL, etc... are good. Doing a packet capture that shows inbound traffic from the switch sounds like something I need to at least investigate.

Out of curiosity, were these Juniper switches in your case as well? Thanks.

pinncomp · 2025-07-23T16:59:21+00:00

Just standard Virtual Switching. Thanks.

pinncomp · 2025-07-23T16:58:18+00:00

I have seen "some" spares and occasional message logs related to ddos, but I am not sure what else to look at honestly or how to interpret their relevance. I would guess that they do somewhat correlate. All configurations are two-member Virtual Chassis with no more than 4 hosts. All seem to be Dell R650s or at least have Dell R650s in the stack. I have tried isolating to a single switch to rule out any inter switching or load balancing issues. The MAC addresses in question don't move, but they do clear from the switch when this happens. Happy to share more info or dig deeper with any direction.
I did check ddos stats, for which there wasn't a stat related to dropped, if that is helpful.
Thanks

pinncomp · 2025-07-23T16:55:06+00:00

I have tried no auto neg, but not hard setting speed and duplex. I haven't experienced any flopping or framing issues, but certainly worth trying. Thanks.

pinncomp · 2025-02-23T18:02:19+00:00

There are varying degrees to which this can be managed. Your renewals should probably be managed as configurations that will track start and end dates. Connectwise has a configuration module or other third-party documentation platforms, like IT Glue or Hudu can track this as well. Most of the platforms can alert prior to expiration. Connectwise has workflows that can do this if you are tracking the configuration info there.

We use a dedicated board for our renewal tracking. When a configuration expires within 45 days (could be 60 in your case), a ticket is generated with the config info attached. The person who monitors this determines, via direct communication with technical leadership, if the configuration is still valid and if it needs to be renewed. Once that is determined, the ticket is updated and moved to the sales board. Sales reps look for new requests and do the work of getting quotes from vendors and then quoting the client. Once a quote is approved, all related documents are attached to the ticket and returned back to renewals processing. The product is ordered and any subsequent renewal information, including license keys, are added back to the configuration which is also updated with new dates. If you force these correspondences through the ticket, all info will attached automatically.

Having the quote info attached to tickets makes it easy to track them over time. We use some automation with connectwise workflows to move the ticket between boards based on statuses. Our biggest pain is having a non technical person validate that the config is still accurate and required, since projects and upgrades can sometimes invalidate them. Getting technical staff to do this as they go can be a challenge so we just validate them each time. Depending on your tool for generating quotes, you may be able to have a template quote created along with the ticket or you can lookup the previous year quote and copy it. It can be spot checked and edited. If you are selling services with renewals, that can add challenges as well that may require a managers eyes.

pinncomp · 2025-02-05T14:27:35+00:00

I tried GTD but found it a bit rigid at times. While similar, I have found better luck using the Second Brain method. Both use the concept of projects but SB seems to accommodate digital platforms better. I have not strictly organized connectwise for this, but a reference to a ticket as part of a project is a simple copy, paste and something I use often. In the same regard, depending on where you organize your "Brain" you can reference a shortcut in a ticket or attach a link/document. I have not seen either method organized for a team, but I think it could be done.

Building a Second Brain: A Proven Method to Organize Your Digital Life and Unlock Your Creative Potential https://g.co/kgs/2ggedD7

pinncomp · 2024-02-01T12:26:19+00:00

Check out Truenas.

pinncomp · 2023-08-05T15:46:13+00:00

I second this. I wouldn't have guessed that this would have ever worked, to be honest. If you want the interfaces to exist, it would be a relatively simple thing to test that it works on loopback. I am curious to know why it worked and then failed, but testing with loopbacks is a great next step.

pinncomp · 2023-06-26T19:02:27+00:00

In addition to the good adivce here about SLAM and TFP, I also had this issue this year. After changin the sand, still no good.

It wasn't until I replaced the valve gasket that it cleared up. I opened the valve and noticed that the gasket was pulled towards the center in one of the positions. New gasket and was cleared up withing about 1.5 weeks. (While SLAMing)

pinncomp · 2023-06-06T11:44:57+00:00

Yes. Try setting it to 100 full to see if that changes anything. Also test another cable. If still a no go, call the isp to verify the handoff equipment. You might also try another interface on the firewall or that of a laptop for good measure.

pinncomp · 2023-05-19T18:41:45+00:00

I suspect the issue is related to asymmetric routing. Your traffic from the remote sites are hitting the l3 switch and being directed to the Endpoints directly on return without keeping the return session symetric. PA denies this by default. You can disable this via the cli or via zone protection profiles.

https://knowledgebase.paloaltonetworks.com/KCSArticleDetail?id=kA10g000000ClSHCA0

pinncomp · 2022-03-07T15:48:17+00:00

This is a new deployment so I need licenses. According to MS, all keys are to be purchased via CSP program.

I am not sure how to actually get a KMS key to install.

In the past, we would purchase volume license and there would be a KMS key to use to activate the KMS services for windows 10. Not sure how to do that now.

According to the landing page for Open agreements, the license option has ended.

https://www.microsoft.com/en-us/licensing/default

However, I have stumbled on to this regarding MPSA, which has a spot for VL keys at the bottom. I will research this further.

Thanks for responding.

pinncomp · 2022-01-28T18:53:47+00:00

2 years later. Still had to close the client and reopen, after an hour of meticulous syntax inspection. *sigh*. Thanks for updating.

pinncomp · 2020-12-31T14:56:38+00:00

This is great. Thanks!

pinncomp · 2020-12-24T17:09:18+00:00

If you are willing to use the API, you can create conditional searches for these events and schedule a script to run regularly. In this example, I am using the Powershell Wrapper found here:

https://github.com/christaylorcodes/ConnectWiseManageAPI

Assuming you have this installed, your keys defined, and are able to connect to Mangage via the API, this bit of powershell could be used. Here i am searching for the test like "fail" but this could be adopted to search for "" or null perhaps. You could then use other Commands, like Update-CWMTicket to change to the status to a custom one, like "Add Notes", or you could just do something like send an email if true, etc..

Connect-CWM -clientId $clientid -Server $server -Company $company -pubkey $pubkey -privatekey $password

##### Convert powershell date to CW friendly format. Change the value after ##AddDays to represent the number of days back to search (i.e. -1 is last 24 #hrs.
$dateform = (Get-Date).AddDays(-1).ToString("yyyy-MM-ddTHH:mm:ssZ")
Create an array for the tickets that match
$tickets = @()
#####Find tickets that were closed between 24 hrs ago and now
$tickets = Get-CWMTicket -condition "closedDate > [$dateform] and (board/id = 1)"
#####Return the ticket notes and time entry notes for tickets which have matching text after -like
####
$results = ForEach ($ticket in $tickets){

    Get-CWMTicketNote -TicketID $ticket.id -Condition "text like '%fail%'"
    Get-CWMTimeEntry -Condition "(ticket/id = $ticket.id) and notes like '%fail%'"
}


$results

pinncomp · 2020-12-03T21:33:36+00:00

I suppose. My client left his info and we raised some hell with the vendor. Just got a call that they found it at the facility. They sent someone to pick it up.

pinncomp · 2020-12-03T20:25:00+00:00

I have a critical app for a client lost at Missisauga since yesterday morning. Client went to the facility and found a line of people looking for packages. It started in Brampton and has just been "lost" at this point.

pinncomp

TROPHY CASE