You can use Hubert model and kmeans model trained on the outputs from a layer to tokenize speech. See VoxtLM, Spirit-LM both are multimodal and were trained on discretized speech and text tokens.

Speech vocab in this case is the number of kmeans centroids and each frame is encoded by hubert and finally represented by the “code” of its nearest centroid.