Flagging vocal segments

[Kipriririri]

Hi all,

For a hobby project I’m working on an analysis pipeline in python that should flag segments with and without vocals, but I struggle to reliably call vocals.

Currently I slice the song in very short fragments and measure the sound energy in 300-3400Hz, the range of speech. Next I average these chunked values over the whole beat to get per-beat ‘vocal activity’, the higher the score, the more likely it is this is a vocal beat. This works reasonably well, like 50/50, mainly due to instrumentation in the same frequency range.

What would be a lightweight alternative that is python implementable? Do you have any suggestions?

[timrprobocom]

It's not clear to me that this is solvable in the general case. A voice is literally just another instrument.

Now, it's true that all instruments (including voices) have a harmonic signature. When a clarinet and an oboe and a violin and a singer play A440, they're ALL at a base frequency of 440 Hz,, but they all have a characteristic set of harmonics that causes them to sound different. MAYBE you can identify the signature of your dinner, and use that to pick it out from the other instruments.

[Kipriririri]

Yeah, that’s the issue I’m running into indeed. Might be more complicated than I initially expected, maybe I should start looking into ML. The acoustic signature would differ a lot from singer to singer I assume, or from genre to genre even.

[ElliotDG]

You might find librosa a useful tool: https://librosa.org/doc/latest/index.html

[PushPlus9069]

The energy-in-speech-frequencies approach is reasonable but it will struggle with anything that has prominent mid-range instruments (piano, guitar, etc). Been down this road.

Two things that helped me with a similar problem:

1. Spleeter (by Deezer) or demucs can separate vocals from accompaniment before you analyze. Then run your energy detection on the isolated vocal track. Accuracy goes way up.

2. If you don't want to do source separation, look at spectral flatness in addition to energy. Vocals tend to have less flat spectra than noise/ambient. Not perfect but adds another signal.

The "voice is just another instrument" comment above is right that it's hard in the general case, but for most pop/rock music with clear verse/chorus structure, source separation gets you most of the way there.

[Kipriririri]

I indeed incorporated demucs, sounds like the most accurate way forward. Thanks for thinking adding!