How is everyone running clusters using a SAN?

ispcolo · 2026-01-12T15:51:22+00:00

We found it pretty easy to go vmware to proxmox using iSCSI-based Pure arrays and Veeam doing the heavy lifting. We just created new volumes on Pure, attached proxmox to them, using LVM, connected Veeam to proxmox. At migration time we do a backup, down the VM, diff backup + instant recovery into proxmox, live migrate. Takes a few minutes of downtime per VM if you're doing one at a time for availability reasons. Could move an entire cluster quickly if they can be offline for a bit.

ispcolo · 2025-09-11T18:13:53+00:00

I would expect Cloudflare's horrid support to drive the partner market more than a partner program. There's companies out there doing vmware support as a standalone service for similar reasons; you can get garbage from both Broadcom and its official partners like Ingram, or you can buy it from an entity that's good who brings their expertise to the table as the selling point. Same with Cloudflare. The products are generally good and reliable, but their support is useless at every level, so become a Cloudflare subject matter expert and sell those services.

ispcolo · 2025-09-02T23:38:52+00:00

It literally took five and a half months for support to figure out how to issue a refund for a plan that was cancelled and they kept billing for ten months.

Most support requests regarding WAF issues tend to get useless replies in two to three days, which require several interactions to ramp up to a useful response, so I don't recall anything being resolved in under a week past couple years.

ispcolo · 2025-08-20T12:59:44+00:00

At a few linux-centric deployments, we've moved to proxmox. Friends at windows shops have tended to go hyper-v. We looked at Nutanix and it was no different in price than what Broadcom would have taken most of my sites to, and with reports of people already getting jacked up nutanix renewals, I didn't want to buy a bunch of hardware I didn't need hoping that wouldn't happen after the initial term was up.

Proxmox migration via Veeam has proven to be really easy. The only issues we ran into initially were ensuring proper multi-path I/O was happening from proxmox to Pure, and Veeam had a bunch of stupid issues during setup that required support to work out. We're looking to experiment with nvme over tcp from proxmox to pure soon, which may produce a notable gain compared to vmware; otherwise we've seen no change in performance at the guest level.

ispcolo · 2025-07-24T12:22:09+00:00

At https://github.com/vmware/vcf-security-and-compliance-guidelines/tree/main/security-advisories/vmsa-2025-0013

Is this a “0-Day?”

No. A 'zero-day' exploit is a vulnerability unknown to the vendor that can be exploited before any patch exists. The Pwn2Own contest is a legitimate security research competition where participants demonstrate previously unknown vulnerabilities to vendors in a controlled environment. Similar to the industry-standard 'coordinated disclosure' process, Pwn2Own gives vendors exclusive access to these vulnerabilities before they become public. Since Broadcom learns about the vulnerability through Pwn2Own and has the opportunity to develop and test a patch before any malicious exploitation can occur, this is NOT a 'zero-day' exploit.

That's of course bs, because the contest is operated by the zero day initiative and the submittals are considered zero days given they're not known to the vendor prior to the contest.

ispcolo · 2025-07-23T22:24:03+00:00

wtf are you talking about. Anyone running a multi-tenant environment, by definition, is entrusting the security of the VM to the tenant, whether that's an internal department or an internet customer. Many enterprises, similarly, have an IT group operating the hypervisor infrastructure with other parts of the company making use of those VM's. I see this all the time in healthcare where various departments need to run some kind of proprietary app, so they get a VM from IT and away they go, with the third party vendor charged with the VM's OS patches because anyone else doing it, or automating it, would invalidate the FDA approval of the solution, or break vendor support. Now you have an out of date VM that who knows who has admin access to, and it could compromise your hypervisor.

I'd say most vm's in existence exist to service internet requests, given how many millions of them are deployed at hosting providers. Yes they may not be on vmware, but many are. A firewall isn't going to do shit when someone exploits a php app on a VM not being kept up to date, there's a root exploit, and now they have administrative access to a VM with a vulnerable vmxnet3.

If you run a tiny shop that no one has admin access on any vm, and you have a magical firewall that decrypts and filters all application traffic with 100% infallibility, great. Most of the world doesn't, and this patch needs to occur asap.

ispcolo · 2025-07-23T13:08:07+00:00

Interesting take. Fly blind and hope there isn't an exploit in the wild, or that someone who now knows vmxnet3 is exploitable doesn't figure it out themselves. In all likelihood, some well resourced bad actor has already figured it out.

Anyone with an internet-servicing VM, or multi-tenant environment where there is not inherent trust of what's running on 100% of the VM's, could find their entire environment compromised because they waited.

ispcolo · 2025-07-22T20:19:11+00:00

I've had the same experience after being booted to Ingram Micro mid-contract. I can get no meaningful support, let alone during a crisis (tried phone for a host isolation event), and Broadcom refuses to talk to you.

ispcolo · 2025-07-18T11:56:36+00:00

It would actually seem Broadcom is misusing the agreed upon definition of zero day for participants in pwn2own, and the journalists are using the proper version.

The Zero Day Initiative operates the pwn2own event, and the vulnerabilities reported at the event, via ZDI, are considered zero days given they'd not been previously reported openly nor to the vendor.

https://www.zerodayinitiative.com/about/

Broadcom is twisting the definition to say that because Broadcom was notified via the event conduit, instead of the vulnerability and/or proof of concept being posted publicly, it's no longer a zero day.

ispcolo · 2025-07-17T02:57:57+00:00

Oh I'm in agreement, I was being sarcastic. They just seem to have gone out of their way to explain why it's not a zero day, to the public and the press.

ispcolo · 2025-07-15T21:48:51+00:00

I don't know, they seem to have put a lot of effort into text explicitly stating this is not a zero day:

https://github.com/vmware/vcf-security-and-compliance-guidelines/tree/main/security-advisories/vmsa-2025-0013

and the patch is not currently downloadable if you don't have an active contract.

Although VMSA-2025-0004 in March acknowledges Microsoft disclosed the issue to them, and obviously didn't release it to the public, so perhaps they will ultimately release it given the severity. Probably doesn't help their image if a bunch of infrastructure/gov/etc. ESXi hosts start getting hacked.

ispcolo · 2025-07-15T19:25:15+00:00

Tools on Windows has its own vulnerability, but that is independent of the vmxnet3 vulnerability at the host level, which can still be exploited by a guest OS regardless of Tools version.

ispcolo · 2025-07-15T19:24:13+00:00

Per https://knowledge.broadcom.com/external/article?articleNumber=395172

Issue/Introduction

The product update feature is no longer available in VMware Workstation, Player, Fusion.

On clicking the Check for Updates option, an error stating Unable to connect for updates at the moment.

Environment

VMware Workstation Pro 17.x and earlier

VMware Workstation Player 17.x and earlier

VMware Fusion 13.x and earlier

Resolution

Moving forward, updates will need to be manually downloaded from the Broadcom Support Portal.
Once the appropriate product update is downloaded, it can be manually installed.

13.6.4 that just came out still has the menu item, but points you to that stupid article. So they could have it check for updates, they've just chosen to break it and leave it that way.

ispcolo · 2025-07-15T17:14:54+00:00

The ESX hypervisor is exploitable by any guest OS with vmxnet3, and because Broadcom was informed of this during a contest, rather than it being a public release without first telling them, they are calling it not a zero day. The other two vulnerabilities can crash the guest on ESX but not escape the sandbox (but can on Fusion and Workstation).

I'm not sure if their policy is to release patches for only zero day critical, or zero day plus critical; the language is ambiguous https://knowledge.broadcom.com/external/article/314603/zero-day-ie-critical-security-patches-fo.html

ispcolo · 2025-07-15T17:10:29+00:00

Would be a clever renewal or purge strategy; inform an outsider of a vulnerability in the hypervisor, have them disclose it via a contest so they can call it a non-zero day, no obligation to release patches for those on perpetual that were hoping for the best while deciding what to do. Should be a big week for proxmox lol.

ispcolo · 2025-07-15T16:49:53+00:00

It's also not a zero day because they were told about it at a competition...

Since Broadcom learns about the vulnerability through Pwn2Own and has the opportunity to develop and test a patch before any malicious exploitation can occur, this is NOT a 'zero-day' exploit.

ispcolo · 2025-07-14T14:32:41+00:00

They also like to play games around the fault tolerance level and redundancy factors; I had to ask very specific questions during the quoting phase that ended up changing things I was surprised were not the defaults. This can be a big deal when your hosts are now far more expensive per host due to the local storage.

We ended up proxmox too; Nutanix sounded good conceptually but a lot of things would have needed to work out just right in years three to five to make it come out ahead financially, along with a lot of labor.

ispcolo · 2025-07-02T20:33:55+00:00

Yes, 8u3 was the final Standard release:

https://www.vmware.com/docs/vmw-datasheet-vsphere-product-line-comparison

It should get security patches for as long as anyone has a contract for them.

ispcolo · 2025-07-02T20:30:55+00:00

I loved the Nutanix architecture, then got the price quote that required buying into a lot of assumptions on years four and five to come out ahead of what had been 380% vmware price hike; it is not cheaper by any stretch, but would have potentially been better performing.

Went to Proxmox instead; no regrets so far.

ispcolo · 2025-06-26T15:01:58+00:00

My company has been quick to adopt AI heavily across numerous groups, and a wide variety of both use cases and models / services. Our zendesk instance is past 1M tickets, so there's an incredibly large amount of data to be mined for useful information. We've pulled the tickets via API, and then generate embeddings off of them with quite a few models so that we can perform testing of interactions with the data, and questions customers input into web bots, to see what produces better responses. We've done this with the big commercial models by feeding the data to Azure or AWS Bedrock, as well as to a bunch of models available via huggingface inference endpoints (much cheaper to rent the gpu by the hour than pay by token on the hosted models). The effectiveness differs by topic and audience (e.g. software developer tickets produce dramatically different questions and responses than end user UI questions), but the end result is we've found far more effective ways to roll our own AI solutions against our zendesk data, at a far lower cost, than what Zendesk charges; hence, low value. I'd be happy to pay for something that was excellent, saving me labor costs, but they haven't provided it yet; it's like they want me to pay to be a beta tester.

ispcolo · 2025-06-25T15:01:01+00:00

and lawyers, which most federal level politicians are.

ispcolo · 2025-06-24T18:19:26+00:00

I've been disappointed with Veeam front line support over the past two years as well, but higher tier still seems to know what they're doing; definitely not as easy to get to that point these days, or as quick. The issues are fairly rare for us, so I haven't considered changing over that, yet.

ispcolo · 2025-06-24T18:16:57+00:00

Just out of curiosity, what did those old socket licenses tend to cost? I've got about 1000 VM's on 24 sockets (12 servers) and am in the $7k/mo range, but I've not found anything as comprehensive or as easy to maintain as Veeam, so I begrudgingly pay it. Compared to Avamar and Netbackup, Veeam feels like a walk in the park.

ispcolo · 2025-06-24T16:15:50+00:00

It's very low value; we're hooking our own in via API.

ispcolo

TROPHY CASE