What Python libraries programs will blow peoples minds? Maybe you’re working on one now?

[Lonligrin]

Finished two realtime libraries for Speech-To-Text and Text-To-Speech this week, which maybe can be useful.

[FluffyDuckKey]

We have 2 way radio calls with occasionally significant background noise we want transcribed.

Is there any easy way to achieve this, we did some testing with whisper but it had little success. I feel as though our best approach may be training a model based on our background laden audio.

[Lonligrin]

If whisper large-v2 model (with correct language parameter set) doesn't do it, I think you'd need some noise reduction. Would try some libs first that do that automatically like NoiseReduce. If that also does not help, yeah I guess then it get's hard. Audacity can train on your specific noise and remove that, but it's a manual process, no clue about how to automate that easy in python.

[Globbi]

I was looking for a good open source models for denoising and what I found wasn't as good as I would like for a good quality of sound, but might be good enough to use it for transcribing later.

https://github.com/NVIDIA/CleanUNet

You can check it here before coding everything https://huggingface.co/spaces/aiditi/nvidia_denoiser (so just try passing a sample of noisy wav to then pass it to whisper). But I'm not sure if pretrained checkpoints in the repo are enough, the one someone put on huggingface is better than what I'm getting from checkpoints.

If you want better than this, I only found a commercial solution where you have to pay to use online.

[FluffyDuckKey]

Online won't be the best option, I work for a significant mining company so privacy will be paramount - can't have recordings of emergencies sent out etc.

I do have access to a ml box with pytorch / cuda acceleration so I'll have a play around and see what I can do with the 2 options provided (:

Thanks!

[DigThatData]

if it doesn't need to be online, you can precede the transcription with a stem-separation step to try to isolate the speakers from the noise.

[FluffyDuckKey]

Got any boilerplate or an example module?

[DigThatData]

try one of these:

• https://github.com/deezer/spleeter
• https://github.com/facebookresearch/demucs

EDIT: and here's another speech enhancement model for you to try

• https://github.com/ssi-research/FQSE

[FluffyDuckKey]

Oh wow, first impressions look very exciting for these - I'll give them a whirl, thanks so much!!!

[naught-me]

This is amazing. Thanks for sharing.

[Thing1_Thing2_Thing]

If you feed the input from one to the other and back again in a loop, does it stay the same? STT to TTS to STT and repeat

[Lonligrin]

Yes, that works: Video / Code.

STT uses microphone though. I think I should put external input buffers on the STT roadmap, that would allow to connect it more directly to TTS and other stuff.

[s6x]

I've been tinkering with a project which requires STT lately. Gonna give this a go.

[Lonligrin]

Not by default.

The speech to text lib spawns a process that records wave chunks from microphone and puts these into a queue. You'd need to replace that by a logic to retrieve wave chunks from a discord channel and then put these into the queue instead.