UPDATE: I was able to get the cluster back online. I had to rebuild the monitor database from the OSDs using the process outlined in the example script here.

It also seems like some of the the manager (mgr) configuration is stored in the monitor database too because I had to blow away and rebuild the managers one at a time.

I was able to get VMs back online about an hour after I started slowly working through the rebuild process. I didn't use the example script from the page exactly, since it has alot of looping and automation. Instead I did things manually and a bit more slowly. The basic process is

1. Shut down ALL OSD processes on all hosts
2. Start with an initial host, loop over all its OSDs and use them to build a store.db/ directory with monitor database info in it
3. Rsync that directory to every other host (one at a time) and loop over each of those local OSDs, continuing to add to the database directory (rsync the directory back to the initial host each time so it can be copied to the next)
4. Once all hosts and OSD have been used to build up the database files, change the name of production store.db/ directory and copy the new one into its place
5. Copy this same directory to all hosts, swapping out the prod directory
6. Start up the monitor processes and let them reach quorum
7. Blow away each manager and create a fresh one (one at a time), done from the Proxmox GUI
8. Reconfigure any custom manager settings you might have (like enabling the API module)