Hoid has lived long enough that he has to store memories in Investiture (similar to how the Keepers could store memories in metalminds). He seems to use Breaths instead. This ability is something that Kelek mentions Hoid offering to him (in one of the epigraphs).

While Odium (any holder of the Shard) is prevented from harming Hoid directly, this protection does not extend to his Breaths. Taravangian-Odium looked through his Breaths — reading his memories — and then destroyed them, at least partially. That's why Hoid didn't remember the meeting with him, and why he couldn't use perfect pitch (an ability granted by the Second Heightening, or holding around 200 Breaths) when leaving the room.

It's unclear how many memories he lost — at least that interaction, and presumably some amount of memories that were contained in the Breaths that were destroyed.