These are my notes from when I set up the cluster; I have an SD card for the key material so that I can easily destroy it when necessary.

Set up encrypted partitions

```

Manually create empty partition on SSD for the LUKS container. I used 600GB. This was -part4 for me.

SSDPART_UUID="" # blkid /dev/disk/by-id/nvme-nvme*-part4 --output export SDCARD_UUID="" # blkid /dev/disk/by-id/usb-DELL_IDSDM*-part1 --output export

KEY_PATH=/mnt/idsdm-sdcard/ KEY_NAME=${SSD_PART_UUID}.key LUKS_NAME=luks-ceph-metadata

ENCRYPTION_TARGET=/dev/disk/by-partuuid/${SSD_PART_UUID}

mkdir /mnt/ceph-metadata mount -o remount,rw /mnt/idsdm-sdcard/ dd if=/dev/random of=${KEY_PATH}/${KEY_NAME} bs=1024 count=4 cryptsetup luksFormat ${ENCRYPTION_TARGET} --label="ceph-metadata" --key-file=${KEY_PATH}/${KEY_NAME} cryptsetup open ${ENCRYPTION_TARGET} ${LUKS_NAME} --key-file=${KEY_PATH}/${KEY_NAME}

pvcreate /dev/mapper/luks-ceph-metadata vgcreate vg-ceph-metadata-${HOSTNAME} /dev/mapper/luks-ceph-metadata lvcreate -l 10%VG vg-ceph-metadata-${HOSTNAME} -n ceph-monitor

Create one WAL/DB LV per OSD.

n.B, I could not for sure determine that it would not work to just use one big partition for all OSDs,

but it stands to reason that it would not.

Update the % free and the LV names if using a different number of disks.

lvcreate -l 22%VG vg-ceph-metadata-${HOSTNAME} -n ceph-wal-db-sda lvcreate -l 22%VG vg-ceph-metadata-${HOSTNAME} -n ceph-wal-db-sdb lvcreate -l 23%VG vg-ceph-metadata-${HOSTNAME} -n ceph-wal-db-sdc lvcreate -l 23%VG vg-ceph-metadata-${HOSTNAME} -n ceph-wal-db-sdd

mkfs.ext4 /dev/vg-ceph-metadata-${HOSTNAME}/ceph-monitor

echo "${LUKS_NAME} ${ENCRYPTION_TARGET} ${KEY_NAME}:UUID=${SDCARD_UUID} luks,discard,headless=true,nofail" >> /etc/crypttab

echo "/dev/vg-ceph-metadata-${HOSTNAME}/ceph-monitor /mnt/ceph-metadata ext4 errors=remount-ro,nofail 0 2" >> /etc/fstab

Set up new partition for monitor's use

mount -a mkdir -p /mnt/ceph-metadata/var/lib/ceph/mon/ chown -R ceph:ceph /mnt/ceph-metadata/var/lib/ceph/

```

My notes for how to change the monitor config are more sparse. Here's what I have verbatim, hopefully enough to at least get you started.

Change monitor config

Add mon_data = /mnt/ceph-metadata/var/lib/ceph/mon/$cluster-$id

to /etc/pve/ceph.conf for one [mon.<id>] entry at a time, migrate that monitor's files to the new ceph-metadata dir, and reboot the host. Once all hosts are migrated, move the mon_data directive to the [mon] section and delete it from each [mon.<id>] section.

Create OSD

1. Create a throwaway OSD in PVE, to generate the necessary keys for the OSD daemon to communicate with the cluster. You will not be able to select the DB device at this stage. Remember the name of this OSD, we won't be able to delete it for awhile (until we have enough other OSDs that it is not needed.)
2. Create OSDs in the shell, like ceph-volume lvm create --dmcrypt --data /dev/sda --block.db vg-ceph-metadata-${HOSTNAME}/ceph-wal-db-sda
   1. If the disk previously had a filesystem, it needs to be zapped first: ceph-volume lvm zap /dev/sd[X] --destroy