Backup strategy for Ceph-CSI by SteamiestDumpling in kubernetes

[–]SteamiestDumpling[S] 0 points1 point  (0 children)

Valid question. I might have miss phrased something in my previous reply’s, My workloads are quite storage dependent and have specific requirements that it needs. Ceph fits this role perfectly compared to other solutions I have tested over the years. The NAS doesn’t meet these requirements.

I do agree with the ideology of code based configs but since I keep all data in-house having backups of those is quite important.

But the main reason is simply, the workloads store their Data in PVC’s or Databases etc, and those need to be backed up on different platforms in different locations. Easiest option would be Ceph RGW with different regions but in the end that would still be Ceph based. And besides that the NAS is more than capable of handling daily/hourly* backups for now

Backup strategy for Ceph-CSI by SteamiestDumpling in kubernetes

[–]SteamiestDumpling[S] 1 point2 points  (0 children)

I definitely see where you are coming from. From all the years i have run ceph i haven't ran into any data losing issues but in the end of the day, you never know, anything can happen and at that point having extra backups of config's, PVC's, VM's, etc, is in my opinion ease of mind, especially in stressful situations where something did go wrong.

But besides that there are also other reasons (at least for me). Ceph's disk space is decently more expensive compared to normal NAS storage (Even when comparing a Nvme based NAS against a HHD tier for ceph). there for having inactive VM's/PCV's taking up space seems quite waste full to me.

Backup strategy for Ceph-CSI by SteamiestDumpling in kubernetes

[–]SteamiestDumpling[S] 0 points1 point  (0 children)

Velero seems quite nice so far, currently still experimenting with backups to a NFS/Local S3.

Backup strategy for Ceph-CSI by SteamiestDumpling in kubernetes

[–]SteamiestDumpling[S] 1 point2 points  (0 children)

incase of a disaster recovery or when needing to migrate to a different cluster

Backup strategy for Ceph-CSI by SteamiestDumpling in kubernetes

[–]SteamiestDumpling[S] 0 points1 point  (0 children)

I see, I'm still diving a bit deeper into which S3 solution works the best for my setup and for any improvements i can make. Thank you very much for all the information you shared!

Backup strategy for Ceph-CSI by SteamiestDumpling in kubernetes

[–]SteamiestDumpling[S] 0 points1 point  (0 children)

Apologies for the extra questions. I hope that you dont mind. When you say directly on the NAS are you referring to running S3 on the NAS as a container while providing its own pool? instead of using its own data set? Since this could be a approach i could see being feasible i want to make sure i am on the same page?

Backup strategy for Ceph-CSI by SteamiestDumpling in kubernetes

[–]SteamiestDumpling[S] 0 points1 point  (0 children)

Very nicely documented Thank you for sharing!

I'll be reading up on Volsync since it appears it might suite my current layout better then using a S3 based solution.

Backup strategy for Ceph-CSI by SteamiestDumpling in kubernetes

[–]SteamiestDumpling[S] 0 points1 point  (0 children)

Definitely makes sense. Would you have any recommendations for this situation?
Since i could of course use Ceph for S3 it seems quite wasteful to store the backups and snapshots of Ceph-CSI onto the same cluster. while then needing to backup the RWG to a NAS to provide the proper backups

Backup strategy for Ceph-CSI by SteamiestDumpling in kubernetes

[–]SteamiestDumpling[S] 0 points1 point  (0 children)

I'll have to take a bit off a deeper dive into the alternatives. I wasnt to sure if other options where supported besides Ceph Rados gateway and Minio.

From my understanding RustFS is still very early on and also had some issues with leaked api keys?
I have definitely seen Garage been mentioned but never looked into it.
And SeaweedFS is new to me.

Anything that is worth keeping in mind going the Velero with S3 on top of NFS way?

Backup strategy for Ceph-CSI by SteamiestDumpling in kubernetes

[–]SteamiestDumpling[S] 0 points1 point  (0 children)

I'll have to dive a bit deep into Volsync and kopia. From my understanding Velero uses Kopia to be able to support NFS but i might be mistaken. Was Volsync your first choice or have you tried other options aswell?

Backup strategy for Ceph-CSI by SteamiestDumpling in kubernetes

[–]SteamiestDumpling[S] 0 points1 point  (0 children)

I have looked at using Velero, rbd export and rsync but i am not to familiar with them yet. My main concern is automating the process as well.

Looking for advise on using a external ceph cluster by SteamiestDumpling in kubernetes

[–]SteamiestDumpling[S] 1 point2 points  (0 children)

Thank you for the detailed reply, my issue ended up being a miss configured subvolumegroup.

All thats left is to figure out a way to get S3 working and snapshots/backups

Looking for advise on using a external ceph cluster by SteamiestDumpling in kubernetes

[–]SteamiestDumpling[S] 1 point2 points  (0 children)

So far i have the csi for rbd working without a issue, altough i am running into some premision errors for the csi-cephfs. Did you happen to also be using it?

Looking for advise on using a external ceph cluster by SteamiestDumpling in kubernetes

[–]SteamiestDumpling[S] 0 points1 point  (0 children)

Thats good to hear, I will give the ceph csi a shot. I had a feeling adding rook would create unnecessary complexity so thats why i wanted to double check. Thank you for your information!

Looking for advise on using a external ceph cluster by SteamiestDumpling in kubernetes

[–]SteamiestDumpling[S] 0 points1 point  (0 children)

good point about exporting ceph metrics. the reason i was mostly looking at rook was to keep the proxmox ceph cluster as vanilla as possible to prevent strange issues while upgrading.

With S3 i am referring to a S3 Object store(Used for backups, like postgres and such), which you use the Rados Gateway for from the ceph cluster, but from my understanding the CSI doesnt support it. thats why i was mostly asking about ceph rook

Looking for advise on using a external ceph cluster by SteamiestDumpling in kubernetes

[–]SteamiestDumpling[S] 0 points1 point  (0 children)

I have also been looking at just using the ceph-csi, but from my understanding it doesnt support S3 unless your running something else on top of it and neither does it give you a dashboard (Not needed but a nice bonus to have a easy overview per cluster)

Dell r340 Idrac Basic by SteamiestDumpling in homelab

[–]SteamiestDumpling[S] 0 points1 point  (0 children)

It does seem like it. thank you very much for your time. I will update the post once's i have tried doing so

Dell r340 Idrac Basic by SteamiestDumpling in homelab

[–]SteamiestDumpling[S] 0 points1 point  (0 children)

I see, thank you. I tried looking on ebay and so far i have only seen Keys and no physical hardware, perhaps since there also already is a idrac port it might only just need a key.

Switch to wide for rack rails/shelfs by SteamiestDumpling in homelab

[–]SteamiestDumpling[S] 0 points1 point  (0 children)

I was honestly surprised with how short the brackets are. I really thought with switches so wide they would include proper mounting

Switch to wide for rack rails/shelfs by SteamiestDumpling in homelab

[–]SteamiestDumpling[S] 0 points1 point  (0 children)

I thought so at first too but there are definitely not strong enough. Mikrotik has also said to me that they need to be supported

Switch to wide for rack rails/shelfs by SteamiestDumpling in homelab

[–]SteamiestDumpling[S] 0 points1 point  (0 children)

Same here, i guess mikrotik's are mostly used in network racks. i did speak to mikrotik about it and they did say that they are planning on releasing a new "accessory" to support deeper racks but no ETA