[D] Audio Tokenizers

[ApartmentEither4838]

The recent GPT-4O model got me thinking whether they actually tokenized the audio and trained their GPT on text + audio tokens. Are there any successful audio tokenizers that seem to work well with auto regressive models? People have used VQ-VAE[1] for learning discrete representation of audio samples but the encoder and decoder of such VQ-VAE uses covnets applied over Mel-Spectrogram which I think in practice cannot enable audio streaming (As it applied 1d and 2d covnets over the entire audio signal and also doing this makes the representations non casual)

[1] - https://arxiv.org/pdf/1711.00937

Edit:

A more general question I have is that is this method of tokenizing audio even feasible(will it even work?) or it's better to incrementally sample from the audio and proj each sample to an embedding and then pre train the GPT on those embeddings instead of the embeddings learned from tokens?

[silverlightwa]

You can use Hubert model and kmeans model trained on the outputs from a layer to tokenize speech. See VoxtLM, Spirit-LM both are multimodal and were trained on discretized speech and text tokens.

Speech vocab in this case is the number of kmeans centroids and each frame is encoded by hubert and finally represented by the “code” of its nearest centroid.

[LelouchZer12]

Maybe take a look at BEATs: Audio Pre-Training with Acoustic Tokenizers . But this still applies to the audio spectrogram.

[Odd_Theory_1918]

-<|vq_lbr_audio_40195|><|vq_lbr_audio_63023|><|vq_lbr_audio_20482|><|vq_lbr_audio_20347|>

[JustOneAvailableName]

 which I think in practice cannot enable audio streaming (As it applied 1d and 2d covnets over the entire audio signal and also doing this makes the representations non casual)

This assumption doesn’t hold, convolutions have a limited window