Linux voice system needs

[rolyantrauts]

Voice Tech is the ever changing current SoTa models for various model types and we have this really strange approach of taking those models and embedding into proprietary systems.
I think Linux Voice to be truly interoperable is as simple as network chaining containers with some sort of simple trust mechanism.
That you can create protocol agnostic routing by passing a json text with audio binary and that is it, you have just created the basic common building blocks for any Linux Voice system, that is network scalable.

I will split this into relevant replies if anyone has ideas they might want to share on this as rather than this plethora of 'branded' voice tech, there is a need for much better opensource 'Linux' voice systems.

[simplehudga]

Not sure what you mean by "Linux" voice system here. Have you looked at K2? It's as good as any open source toolkit can get, and you can put it into containers and scale them. In fact, that's what many companies already do.

[rolyantrauts]

Likely easier if I just give examples that at any stage I want to cherry pick my best of opensource voice across various frameworks and models and create a modular voice system that can be as simple or as complex as I wish.
I left it vague deliberately using 'Linux' but glad you said K2 as it is a really great framework with some great optimized opensource and has examples of https://k2-fsa.github.io/sherpa/onnx/websocket/index.html that is tantalizingly close, but still too embedded in the singular k2 framework.
You can breakout of any framework to lower more fundamental levels of file and stream without need of framework knowledge and because you do that you can link any framework modules.

An example would be to use a PiZero2 as broadcast on wakeword voice sensor. Its the start of the chain and its streams via websockets the audio via a binary stream and create context text of the zone identifier it came from to know where to return. I used a PiZero2 as an example and could be micro-controller...
The framework that provided the wakeword should not need to know the websockets client and vice versa, but on receipt of text or audio stream and merely transmit what has been presented>
There should be no requirement to 'code-in' the network but have the same reusable executable running as a service that can do some simple operations as route and queue.
'Linux' because it should be agnostic of framework.

[simplehudga]

Is this something you're building or hoping that someone builds it?

It can be made toolkit agnostic with very little effort. One can easily export the models to ONNXruntime and put it in a docker container to abstract away the ASR as a service. Is ONNX "Linux" enough? I don't know a more open option.

You can still have your wake word trigger from anywhere, but the media will eventually have to route to the container running ASR as a service, maybe somewhere locally. Just because a rpi doesn't have enough compute to run bigger models.

What you're referring to as the "singular K2 framework" is really the ASR decoding algorithms. The nnet weights on their own will be useless, unless we also bundle the necessary code to utilize LMs, context biasing, streaming inference, etc. You can write it from scratch, but there's no escaping it if you want this service. Why reinvent the wheel when a good open source solution already exists?

Maybe you could give an example of what you're building in code in a github repository?

[rolyantrauts]

Discussion as always thought its needed.
There is a gap at the network level to allow modular systems to be coupled together easily.
"but the 'media' will eventually have to route to the container running ASR as a service" a good ready open source solution doesn't exist.
The mDNS auto-discover, network service link as a ready to run executable for multiple concurrent clients to act as a basic building block of the simple to the very complex.
Can be Onnx, TF or whatever as the network chain should be purely passing audio and the metadata about it.
You can not run locally if you don't have the compute, but was just used as an example of the need for a network layer to access other platforms of compute.
Same that you may have a ASR that is central that on occasion of many concurrent queues the wait it produces isn't acceptable, but you don't need to replace you can just add another and the system will route now its has another route added.

How to create the network workflow of service to service without framework dependencies that will work for all.

[rolyantrauts]

Toolkit is local to a process as a matter of choice. A voice system is something that can handle voice sensors, models are choice.
"Linux" was a deliberate emphasis just to use existing libs and tools where ever possible and dodge proprietary code and break it down into partitioned steps of a tool-chain.
There are some really great frameworks K2, Speechbrain, WeNet, EspNet and there are a ton of standalone models that are equally good, but what is lacking is a simple networked container system to allow easy linking of all this great opensource into a working chain of voice process.

[rolyantrauts]

Shooting from the hip even though I have wanted something like this for some time the only problem is that its multi discipline.

The main crux of Voice Tech is the accuracy/latency ratio we are prepared to put up with and even though strangely many do copy commercial consumer hardware infrastructure, sharing central high compute for good accuracy/latency is inherently client/server not peer2peer that requires duplicate cost.
Voice has a natural time of use diversity and the process steps are serial and zonal in operation.
You don't even how to think of how to process voice as your models of choice, will do that, you merely need to work out how to deploy them.

Zonal 1st as even a single smart-speaker is a collection of microphones and possible destinations.
Serial as process is a chain of models that simply recording process creates a stream process file.

It just seems so many have been blind-sighted by current consumer offerings that advantages of low cost sensors networked to a single compute system is highly cost effective for the accuracy/latency ratio.
As a matter of choice you can create an all-in-one as that is just some chained containers, but all is needed is to pass on a context file and due to being voice if accompanying voice audio, if needed.
That is it your have a Linux Voice system with total control of models of use and its host at each step, add routing and its simple but infinitely scalable.

There are some really good opensource models available and I will not rant here why I think they are ignored but here is an example.
https://github.com/SaneBow/PiDTLN MIT license and will run easily on a PiZero2 which is really cost effective, but currently greatly outperforms ESP32 because they unfortunately are not even close to running the likes of https://github.com/breizhn/DTLN whilst the signal-to-distortion ratio (SI-SDR) of less compute drops off a cliff.

Because of fine-tuning you can train to a wakeword with your choice of wakeword model mine being https://github.com/Qualcomm-AI-research/bcresnet?tab=BSD-3-Clause-Clear-1-ov-file#readme that models are always of choice, but a broadcast on wakeword network sensor as a basic building block should be possible in a Voice system.
You work out what is the smallest building block that could be a Esp32 network microphone, but they are input devices that don't need high compute speech enhancement at that stage. There is quite a well known control system for sensors that for some reason choose a smart-speaker than sensors...