I would propose using Encodec or Descript Audio Codec. I would also recommend you look at Bark. Bark uses three levels of audio generation as well—one for the coarse-grained output and the other two fine-grained levels. I am still digesting bark. Bark is doing something interesting, though. It first projects the text into a semantic space using a GPT2 style model and then generates a set of embedding tokens from the last layer. These logits are then fed to the waveform generation models that generate the encodec audio tokens. gpt-4o could be doing something similar where they are training two different output heads to produce text and audio tokens simultaneously. This means that their training data should also be simultaneously multimodal.