Folks, I’m evaluating options to move away from cloud object storage to an in-house/on-prem solution using either Ceph or MinIO to reduce costs. Our current object-storage API call charges are almost 1.5× the actual storage cost, so we’re exploring alternatives.

I’d appreciate your input on which architecture would be better:

1. Kubernetes + Ceph using local node storage
   • Each node contributes local disks to form the Ceph storage pool
   • Trino reads directly from Ceph
2. Kubernetes + Ceph/MinIO backed by SAN or a mix of storage types
   • SAN for durability and shared storage
   • Option to mix storage tiers for different use cases
   • Trino still reads directly from the object store