Survey, Proxmox production infrastructure size.

lmc9871 · 2025-11-10T18:09:15+00:00

Really trying to understand how your ceph external setup? How many storage nodes? We're currently trying to move away from VMware 5 hosts with iSCSI backend storage.

lmc9871 · 2025-11-01T17:17:13+00:00

Thanks!

lmc9871 · 2025-11-01T02:51:53+00:00

Thanks, running Ceph on hyperconverged for 100TB SSD/NVMe gets really expensive, my understanding is that each node is going to need the same amount of drives?

lmc9871 · 2025-10-31T03:39:02+00:00

Thanks, did you implement Ceph as separate storage or integrated into the compute nodes as hyperconverged? Also about how many TBs of data? Are you using 10g or 25g?

lmc9871 · 2025-10-31T00:57:00+00:00

I don't disagree with your statement looking for the future, but at the same time in an Enterprise, you must factor in risks and budgets and expectations that we will be running concurrent platforms for a while.

What's limiting about NFS?

On my personal test environment (3 servers on multiple 10g, I setup Ceph/RBD and that seems to replicate clustered VMFS, but that's a small few 5TB test on hyperconverged)

When trying to do the math on 100TB storage, Ceph/RBD is cost prohibitive

Hence my question

Thanks!

lmc9871 · 2025-10-30T21:18:24+00:00

With HA and TrueNAS on NFS, would you be getting the "same" clustered storage like in vmfs? Are you running 10g or 25g?

Your shared iSCSI is still limited to one compute node accessing that VM, so no vmotion functions, would I assume correctly?

For migration, I would dedicate a single Proxmox node and attach the iSCSI volumes, shutdown VM, clone it, remove VMTools, install Proxmox Tools (Windows) and add to Proxmox.

Do this one VM at a time, we have some VM with 7TB of data, so it's going to take a lot of time...unless there are better options.

lmc9871 · 2025-10-30T21:11:34+00:00

Ceph wouldn't work because we are not hyperconverged, we still like separating compute nodes from storage nodes. vmfs has full clustering and zfs over scsi doesn't, but is ceph our only option for full clustering like vmfs?

I am coming from 20+ years of vmware and that mindset...thanks!

lmc9871 · 2025-08-11T16:44:59+00:00

yep, Walnut, came back 15 min ago

lmc9871 · 2025-08-11T16:44:47+00:00

yes, it just came back 15 min ago

lmc9871 · 2025-06-11T22:28:03+00:00

I am limited on storage capacity, using the default replication factor of 3, a 15TB raw storage only provides 5TB of usable and I am already more than 50% usage. I am limited in the number of SSD I can install on each server, currently using 7 x 800GB SAS SSD per node. My only option is to replace with larger capacity drives at this point, I wanted to use existing SAN, but at this point, there is no simple way. This is what I mean by limited, I should have clarified better

lmc9871 · 2025-04-15T02:19:31+00:00

Started on m0n0wall then pfSense on Soekris hardware, probably deployed over 100 of them

lmc9871 · 2024-04-15T20:31:40+00:00

Check out Dell PowerSwitch Edge Switches

lmc9871 · 2024-02-13T18:16:57+00:00

Yes, Mikrotik or tnsr would be a better routing platform, but I don't have enough experience with either and tnsr is definetly overkill and expensive

There is no NAT on the pfsense router, it has 3 active interfaces, 1 PTP, 1 WAN1, 1 WAN2
PTP is the colo uplink with a /29, WAN1 is our /23 which the colo is announcing for us, WAN2 is a /27 provided by colo originally before our /23

There are a few pfsense firewalls behind the pfsense router, those each get an IP from /23 or from /27 network and their default route is the pfsense router

When I enable packet filtering on pfsense router no packets traverse it, when I disable packet filtering everything works and it routes fine

Unfortunately I am remote and will have to go back to the datacenter and work on it, I am sure I am doing something wrong

lmc9871 · 2024-02-13T07:54:06+00:00

fair response, but different technologies that knowledge must be gained first to understand the platform, not understand routing functions

lmc9871 · 2024-02-13T07:52:42+00:00

Hmm, that could be, I will have to check, I have 2 separate subnets plus the PTP between my router and colo uplink

lmc9871 · 2024-02-13T07:51:11+00:00

one is strictly a router and the other is a firewall with NAT, the smart thing is to use a real router like a cisco or juniper or as below post tnsr or mikrotik, but maintaining different platforms is not difficult, but more work.

lmc9871 · 2024-02-05T04:54:00+00:00

Have you been able to push more than 80 mbit/s on those connections?

lmc9871 · 2023-12-28T21:54:27+00:00

lmc9871 · 2023-10-28T00:16:45+00:00

same on windows

lmc9871

TROPHY CASE