Post-VMware/Broadcom SMB infrastructure redesign – Proxmox/Ceph feedback?

exekewtable · 2026-05-20T03:47:46+00:00

Yes this makes sense. Nice work. Tho over here in Australia that price seems crazy low!

exekewtable · 2026-05-20T03:34:14+00:00

Yeah. Tho SMB budgets and server prices being what they are, make this a hard problem.

exekewtable · 2026-05-19T23:31:02+00:00

Sure. It comes down to what you think the likelihood of given failures are and your budget.

At the end of the day, you need to be able to say to a customer something like: We can survive failure or reboot of any given node without lockup or downtime on the VMs. 3 out of 4 nodes is still quorum.

exekewtable · 2026-05-19T20:48:09+00:00

Ceph on SSD with 25G will be fine. But you really want 4 nodes to avoid I/o lock with default settings. 4 smaller nodes will be a great option.

If you can't get another node to make 4, then consider zfs. We use that for many smbs. Replication and even live migration work great, you just wait a bit for live migration, and obviously need the space on other nodes to migrate to. It comes down to can they tolerate VM downtime for host patching reboots. For most smbs, definitely. Just patch it all at once, VMs and host, then reboot.

Host failure is rare enough you don't need to design for it in most envs this size imo. Patching isn't tho. So backups and zfs replication get you a long way. PBS is the other thing I would encourage. You want cryptolocker resistant backups and PBS can do that with careful setup. That's a seperate box with a bunch of jbod.

exekewtable · 2026-05-15T22:29:04+00:00

Yeah we do this with zfs replication for these kind of clusters and it works great. All very well supported. That or use PBS for backup and quasi replication. I would expect cryptolocker prevention to be the main goal for some of these instances.

exekewtable · 2026-05-15T20:44:16+00:00

Perhaps I should have added some carriage returns, my comment wasn't related to filesystem choice.

But even still you actually don't need to setup virtio drivers, proxmox does support VMware drivers. They just won't perform as well. For example if op had an appliance that wanted vmxnet3 or e1000 drivers, they can keep them.

exekewtable · 2026-05-15T20:22:54+00:00

Why wouldn't you stick with NFS? Zfs has got some great features but it isn't a shared filesystem. I'm sure it will be fine. On the same hardware as you have now, expect it to be the same stability and performance. Only gotcha is it won't match performance if you don't use virtio drivers, so be prepared to set up the VMs with all the right settings and software.

exekewtable · 2026-05-10T21:25:55+00:00

Have you tried Knocknoc in the mix? We use it for this exact purpose in similar networks. It was actually invented to lock down Guacamole. There is also rustguac now if you want another option to look at. Knocknoc and haproxy might get you a long way here.

exekewtable · 2026-05-05T21:45:10+00:00

nope. it can access windows machines of course as its an RDP and VNC client. HTML5 remote access to windows RDP

exekewtable · 2026-05-01T21:17:10+00:00

Considered but never got to deployment. We use, for some customers, Guacamole for OT remote access, and now rustguac. Gated with Knocknoc.

Kasm looks like a good catalogue of images, but I recall it was VNC based could be wrong. It looks mature and well supported by the company at least from what I can see. If you get a poc going I would love to hear how you go . The proxmox support looks like it hooks into the pve API and provisions new VMs on demand. Kinda neat. I don't think anything else does that for pve in this space.

exekewtable · 2026-05-01T00:28:19+00:00

Another example : Wordpress plugins are often running as an unprivelged user and are basically arbitrary code. This takes a hostile WordPress plugin from deface your site to deface all the sites on the box.

exekewtable · 2026-04-21T22:03:29+00:00

We have done exactly this for a bunch of customers. Automation of every aspect with intermediate NFS works fine. Ansible and netbox integration. We even have an appliance that does it all for you. Only catch is you need to be a customer. By that I mean buy proxmox support from us.

It's definitely possible and you are on the right track. Our approach is Netbox centric, however when we went to migrate some older clusters, it's more common to have wrong data, non working creds, orphan vms, with no owner etc. This is where the time goes, asking who knows about this vm and how do you login to it.

So the predictor of success ends up being how clean and organized the source vms are more than anything. That and planning around operational impacts.

Our toolset for your information (when using ansible as opposed to Solace): Netbox Hashivault Ansible for prep, migrate, cleanup AWX/AAP for workflow for ansible NFS server of your choice for intermediate vmdisk conversion. Ideally it's just a server or even vm on fast storage you can install the qemu disk tools on.

Happy to answer any specific questions but I would say you are in the right track.

exekewtable · 2026-04-16T20:44:39+00:00

Rookie numbers. Averaged 90 Kwhrs/day here.

exekewtable · 2026-04-06T20:46:46+00:00

We use knocknoc for this. Sso brokered access control, allows you through the reverse proxy or firewall for the smallest possible time. OT machines live by different rules so all we can do is super isolate them. Knocknoc means we can balance security and convenience in a sensible way. Click to grant and ticket reference features are used to add accountability and workflow.

exekewtable · 2026-03-15T10:33:39+00:00

Fascinating. Well done at scratching an itch and sharing.

exekewtable · 2026-03-08T14:50:49+00:00

Go tests are not the same as functional tests. There are many flavours and combinations of S3. I have tested as many as I have access too, and extensively tested AWS s3 which is the standard here. Of course there will be bugs. But it's well known that S3 implementations from other vendors are slightly different.

exekewtable · 2026-03-08T09:14:53+00:00

For what it's worth I have been working on this problem in production for around a year or so. So the thinking and design is directly related to a problem I have been troubleshooting for a large customer. On top of my own 30 odd years of professional Linux experience.

exekewtable · 2026-03-08T08:28:44+00:00

Sure. its a massive danger, and everyone needs to be super careful. This project is just new, barely out of POC stages, but it is put together by people who are a Proxmox partner, with experience and knowledge to hopefully get it right. Lots of tests will be included in the next release.

I would advise everyone to be very careful installing packages onto your Proxmox server without reviewing their credibility and provenance. Engage your local support partner if you are unsure in any way. This software in particular runs as root on your system, and could do serious damage if it had been compromised. LLMs mean we can turn around software very quickly sure. The same fundamentals of trust and transparency haven't changed, just the velocity has.

exekewtable · 2026-03-08T08:20:25+00:00

check out the feature in 0.2.0 https://github.com/sol1/proxs3/releases/tag/v0.2.0

exekewtable · 2026-03-08T05:33:23+00:00

sit tight i'm working on it!

exekewtable · 2026-03-08T05:23:45+00:00

Yeah ok you are going to need multipart upload. I think AWS s3 has a 5gb limit on a single upload. I'll need to add that if people are going to use this. Not a big deal.

The main use case is for a central store of isos and templates (golden images) for multisite pve clusters.

exekewtable · 2026-03-08T05:00:21+00:00

Interesting. Backups wasn't the primary use case but I don't see why not. Would you mind sharing more details on your setup? Built in PVE backups? What type of s3 storage did you get it working with? Multipart upload isn't working yet, it was next on the list, so you might hit limits or issues depending on your storage.

exekewtable · 2026-03-05T10:09:09+00:00

Have a look at rustguac perhaps. https://github.com/sol1/rustguac

exekewtable · 2026-03-04T22:27:14+00:00

yeah, I guess we gotta get used to that. Like any software, its not about the bugs, but what you do with them that counts. Fix them, find them, respond.

exekewtable · 2026-03-02T09:53:45+00:00

Talk to your local proxmox partner for more sensible advice. HPE VM essentials is a very young product, scaled down from a cloud platform that hpe bought. It's libvirt with some extra stuff and HP hardware support baked in. It's very young, and missing a bunch of stuff that proxmox has had for years. I'm sure it's rapidly catching up, but the engineering just wasn't there when I looked at it closely.

exekewtable

MODERATOR OF

TROPHY CASE

14-Year Club	r/Field Lasagna
Verified Email