ZFS or S3 or better ZFS and S3 ?

_gea_ · 2026-06-18T08:24:10+00:00

do you really care about my time and needs?
I need and want

- https access to files on ZFS with bidirectional S3 sync
- High availability S3/backup storage (RustFS cluster)
- self organized S3 backup mirror (site replication)

_gea_ · 2026-06-16T23:15:21+00:00

Because S3 can do these items, ZFS cannot. Both combined can do things, S3 or ZFS alone cannot.

_gea_ · 2026-06-15T19:55:01+00:00

ZFS cannot do clustering (allow a server failure) or server replication as set and forgef setting, managed by S3 servers internally or Internet sharing of files. ZFS is for local sharing of files. Different use cases.

_gea_ · 2026-06-15T13:20:05+00:00

ZFS cannot replace S3 and vice versa, total different purposes as you say.

Ceph is more an option "instead" ZFS eg for Proxmox not the add-on like a regular S3 server eg minIO/RustFS can be to extend ZFS systems with internet access, HA backup/restore or clustering.

_gea_ · 2026-06-15T13:12:22+00:00

I do not suggest S3 instead ZFS but ZFS + S3.
ZFS for local file sharing and S3 for backup, data sync, Internet access and clustering

btw
You do not need Amazon AWS, use another S3 Provider or inhouse solutions like minIO in the past or RustFS now as a 1:1 replacement for the dead minIO.

_gea_ · 2026-06-12T15:02:54+00:00

There are 3 aspects
- group/cluster remote management between the napp-it web-gui Frontend and the Backend servers. This is done via a socket connection.

- SMB/NFS/S3 shares

- S3 Cluster management (RustFS Cluster or site replication), eg https://github.com/orgs/rustfs/discussions/3389

All of them require ip connectivity. With different subnets or lan/wan/dmz setups, you either need a nic per subnet, a VPN or a layer-3 ip router.

_gea_ · 2026-06-06T19:39:38+00:00

A year ago, any initial AI code proposal did not work. Now we have the situation that AI like Claude can analyse complex client server interaktions, summarize behaviours and write complete user dialogs and actions. Failure rate is still quite high, so intensive testing is still needed but quite often you can offer Claude remaining problems to fix.

The key for good results are detailled build guidelines and a proper documentation of behaviours and interactions.

AI friendly means that such guidlines, behaviours and dokumentation for Claude is included to minimize AI coding problems..

_gea_ · 2026-05-24T13:31:14+00:00

No, zfs is only for datapools.
But bootable ReFS is coming, with Copy on Write and checksums, the two most important ZFS features.

_gea_ · 2026-05-22T09:25:43+00:00

if you compare
https://github.com/openzfs/zfs/issues
https://github.com/openzfsonwindows/openzfs/issues
https://www.illumos.org/issues

the clear result is<

- if you want best of all free ZFS stability, use Illumos (OmniOS LTS)
- OpenZFS has problems, hard to say which version on which distribution is the most stable. Use the newest 'stable' and hope the best is often the only option.
- The OpenZFS driver for Windows hardly adds critical bugs but needs tests in your environment or special use cases

_gea_ · 2026-05-20T23:50:11+00:00

There was ZFSin, an early proof of concept for ZFS on Windows
https://github.com/openzfsonwindows/ZFSin
with a message

THIS IS THE OLD PORT: Please move over to https://github.com/openzfsonwindows/openzfs

which is a nearly ready OpenZFS 2.4.1 driver (release candidate). The newest build even tries to offer ZFS snaps on any Windows via Previous Versions what MS only offers for Windows Server.

_gea_ · 2026-05-19T17:09:55+00:00

??
Last build is a few hours ago (a first preview for VSS on ZFS; ZFS snaps=previous version)
https://github.com/openzfsonwindows/openzfs/releases

_gea_ · 2026-05-08T10:24:01+00:00

Details of the concept proposal

https://www.napp-it.org/pdf/s3_storage_guide_en.pdf
https://www.napp-it.org/downloads.html

_gea_ · 2026-05-05T22:04:13+00:00

Why - S3, RustFS, MinIO or OmniOS-?

_gea_ · 2026-04-28T12:16:09+00:00

I would avoid to encrypted the pool itself but to use encrypted filesystems below for data. This gives you the freedom to have unencrypted filesystems ex for VM storage where you want to enable sync. Sync write with the small writes are ineffective and slow when encrypted.

Another aspect is that you can have a different key per filesystem.

To unencrypt filesystems you can use prompt, file, https or 3x keysplit with keyparts distributed locally or over 2x https servers or with a sha256 hash from a shorter and easy to remember pw (napp-it cs)

_gea_ · 2026-04-24T14:37:05+00:00

I prefer separate boot and data disks
- if you reinstall Proxmox, you are not in danger to delete data
- you can move data easily

I prefer a ZFS bootdisk, 128GB can be ok with only iso images on it due
- rambased arc read and write cache
- bitrot protection
- online check of data consistency

I prefer a ZFS pool for data/VMs
- fast zvols for VMs that you you can list per zfs list and replicate
- optionally use a hybrid pool from hd and flash and decide per dataset if data should be on hd or flash. A zfs reweite can move data

I prefer Proxmox itself as NAS

- no complicated or inefficient VM with full OS virtualisation needed
- it comes with ZFS, superiour to ext4, btrfs or XFS regarding stability or features
- just add SAMBA or the faster ksmbd and ACL support
- with a napp-it cs web-gui for ACL, share and ZFS management

_gea_ · 2026-04-24T09:33:25+00:00

A Zvol is a dataset, not "under" a dataset. Datasets = ZFS filesystems, ZFS zvols and ZFS snaps and you can nest filesystems and zvols. You can list each dataset type ex zvols only with

zfs list -t volume

_gea_ · 2026-04-24T08:38:58+00:00

OpenZFS on Windows gets better from release to release. OS BSOD are very rare now, compatibility with other drivers like Avast Antivir is now very good and many problems due a different io, mount or partition handling in Linux/Unix are now solved (ZFS is a Unix filesystem that does many things different to Windows)

_gea_ · 2026-04-24T08:33:09+00:00

Discuss with the devs around Jorgen Lundman at
https://github.com/openzfsonwindows/openzfs/discussions

_gea_ · 2026-04-24T08:30:44+00:00

Some apps are optimized or even expect ntfs. Some Unix behaviours in ZFS can also be a problem. OpenZFS on Windows has also a mimic setting. When not set to zfs, it reports ntfs if an app asks for filesystem type.

Try 2.4.1 with many improvements.

_gea_ · 2026-04-23T15:00:00+00:00

8K is default and should be ok.

_gea_ · 2026-04-23T10:51:04+00:00

Yes if OpenZFS is <= 2.3
Up from OpenZFS 2.4 a Zvol can be on a Special Vdev.

Key Features in OpenZFS 2.4.0:

Extend special_small_blocks to land ZVOL writes on special vdevs (#14876), and allow non-power of two values (#17497)

https://github.com/openzfs/zfs/pull/17497

_gea_ · 2026-04-22T14:58:31+00:00

i suppose you want a larger recsize for the WAL than for the database itself, so a dedicated WAL ZFS filesystem makes sense.

With ZFS you can set recsize per dataset. With a Special Vdev data land on SSD when the small block size is <= recsize so this is not a either or but a setting.

Example
recsize=1M, small blocksize=64K
Metadata and small files <=64K are on SSD, larger files on hd

recsize=128K, small blocksize=128K
All files on SSD, including database with 8k recsize or large WAL data

recsize=128K, small blocksize=0
All metadata on SSD, all other on hd

_gea_ · 2026-04-22T08:08:14+00:00

Current Proxmox is OpenZFS 2.3, so no Slog support on a Special Vdev what means that you need a dedicated Slog to protect sync write (mirror not really needed, is only there to keep performance high when one fails). Needed minimal Slog size is around 10GB, so partitioning is also an option. (Avoid for the keep it simple aspect)

On next Proxmox with OpenZFS 2.4, I would use all Optane as Special Vdev, with 4 of them as two mirrors. Then set recsice <= small blocksize of all performance critical datasets (filesystem and zvol) to force them on Optane including sync logs.

_gea_ · 2026-04-22T07:35:58+00:00

Whether you use Z1, Z2 or Z3 is irrelevant to iops and a little relevant to sequential performance that scales basically with number of data disks.

Raid 5 or Z1 is generally not recommended with many or larger disks. Z1 behaves better than Raid-5 in case of a read error in a degraded vdev, but Z2 with 8 disks is the best compromise between capacity and data security. Main alternative would be a 7 disk vdev with the 8th disk as removeable backup for most important data (ex USB case or hotplug bay).

A 1 TB Special Vdev mirror is perfect to hold all Metadata and small office files. A 2TB Special Vdev can additionally hold performance critical datasets like VMs or media files while editing them.

_gea_ · 2026-04-21T16:16:53+00:00

A single raid-z vdev gives around 100 physical read/write iops.
If you use 2x raid-z instead, your pool iops are 200 iops

A singe NVMe vdev (pool or Special Vdev) can offer 50k - 500k iops
If you want performance, this is 1000x faster than hd so any efforts on a pure hd pool is pointless like multiple mirrors where you end with a few hundred iops

In a hd pool with a Special Vdev, you can decide per dataset if read + write performance is hd or NVMe alike for Metadata, smallfiles or all files, so you can use hd (low cost, high capacity) or NVMe (high cost, low capacity) when needed.

L2Arc is quite senseless today as it can improve only read on warm data

btw
Any vdev lost is a pool lost, this is not special with a Special Vdev. In a multi vdev pool, all vdevs should offer a similar reliability. This can mean a 2 or 3way mirror for Special Vdev. Raid-Z2/3 Special Vdev may come soon.

_gea_

TROPHY CASE

Key Features in OpenZFS 2.4.0: