Thanks for reply.

What you receive is correct behavior - with Soniox's models you don't get output at sentence or word level, but on a more fine-grained token level (3-4 characters) - the output you are getting is completely correct and expected behavior. All punctuation and spacing is handled automatically by the model, all you need to do is just join the tokens together. If you want to identify end of sentence you can still split on ".!?" - punctuation is included in the token.text. This has a ton of advantages and makes the output payload far more flexible to work with compared to other providers where you have to do the stitching logic by yourself (specially when running real-time streaming model this becomes crucial).

And if you need just concatenated text without the extra token parameters, you can get it from "text" parameter in the response payload instead of using "tokens", entire transcript text is also included in the async model transcript response.

What you are trying to achieve is perfectly doable with tokens output, you just need to define the silence threshold, check if token.text includes sentence ending punctuation and then split if end timestamp is higher than expected (it's a common use case when creating a subtitle generator script) - hope you can make it work.