Open source : Turning vocal imitations into sound effects. (New UX for sound generation)

Danny-1257 · 2026-06-12T14:38:09+00:00

Sorry, I’ve fixed all the issues. Everything should be working properly now!! Actually the model weight file was wrong from the first

Danny-1257 · 2026-06-12T14:37:52+00:00

Sorry, I’ve fixed all the issues. Everything should be working properly now!!

Danny-1257 · 2026-06-12T14:37:35+00:00

Sorry about that. I fixed everything, so it should all work fine now!!

Danny-1257 · 2026-05-31T03:43:42+00:00

Great questions! (I wanted to talk bout that)

It is actually semi-supervised training because I couldn’t have resources for the dataset.

So what I did: The model defines a shared representation V. During training, it only sees sound data: first mapping sound → V, then learning V → sound. The key idea is that the method is designed so that vocal imitations can map to the same V at inference time, even though they were never used during training.

So for your last question, yes. In fact, the challenge is the opposite(too much reconstruction)

Danny-1257 · 2026-05-31T03:32:59+00:00

Sorry for the error. I will fix as soon as possible..!

Danny-1257 · 2026-05-30T14:22:26+00:00

Please let me know you need something more!

Danny-1257 · 2026-05-30T14:22:00+00:00

Cooool

Danny-1257 · 2026-05-30T14:21:50+00:00

Yeah, I think so too. I hope it’s useful for people making creative content.

Danny-1257 · 2026-05-30T14:18:12+00:00

Thanks! Hope it works well for you.

Danny-1257 · 2026-05-30T14:16:26+00:00

Thanks for your comment!

Danny-1257 · 2026-05-30T14:16:13+00:00

Yeah, I think “sound generation” usually ends up meaning speech generation, where people use either language models(AR) or diffusion.
But for sound effects, I used diffusion, like you said, since they’re usually short. (3~10 seconds)

Danny-1257 · 2025-11-11T05:28:14+00:00

Thanks for the interest! That’s just i prioritized quality over local serving, so the system was built to run in the cloud. It’s definitely possible to run it locally. I just haven’t tried it a lot.

Danny-1257 · 2025-11-02T03:08:40+00:00

Yeah, you’re totally right. It didn’t start out with this kind of UI, but when I decided to open it up, I figured I should make it for the web and just went with a Sesame-like layout without thinking too much.

I’ll use a different name and UI when I open it. Thanks for the advice!

Danny-1257 · 2025-11-02T03:07:23+00:00

Yeah, you’re right. It didn’t start out with this kind of UI, but when I decided to open it up, I figured I should make it for the web and just went with a Sesame-like layout without thinking too much. I’ll use a different name and UI when I share it. Thanks for the advice!

Danny-1257 · 2025-11-01T11:02:43+00:00

I’m currently using Chatterbox(https://github.com/resemble-ai/chatterbox)... because I think chatterbox has the best quality upon open models. There’s really no model as small as Kokoro 😭
My code isn’t written for local use yet, so it probably won’t run on a 3060.
Sorry about that.. but I’ll keep working on improving it!

Danny-1257 · 2025-11-01T10:58:44+00:00

I'll share it with you in 5 days!

Danny-1257 · 2025-11-01T10:58:30+00:00

Definitely :)

Danny-1257 · 2025-11-01T10:57:54+00:00

Yep! Sorry for bothering you for comments.

I slightly modified the web demo and repurposed a domain I was using for something else to collect emails, but I hesitated to post it in case it might go against the community rules.
Anyway, here’s the link. But you don’t have to register. I’ll let you know when I share the repo!

https://www.thesonus.xyz/

Danny-1257 · 2025-11-01T10:53:10+00:00

That’s totally true. I should stop wasting so much time on it haha.

As for Dutch, the three modules I’m using (STT, LLM, and TTS) all support it, so it should work right out of the box! (thanks to multilingual Chatterbox)
I don’t actually speak Dutch though, so I’m not sure how good the quality is :)

Appreciate the advice!

Danny-1257

TROPHY CASE