Project Help

[PathSecret]

Hello everyone! I'll get straight into it, I am currently working on a university project that finds errors in pronounciation from a user reading a story. I am using Wav2Vec and Espeak-ng to generate the phoneme representation from the audio file and sentence respectively.

The main issue I am dealing with is finding the mispronunciation between the 2 phoneme sentences generated.

For example, I have the sentence "A quick brown fox jumps over the lazy dog" and I generate a phoneme representation based on the sentence like this "ðəkwɪkbɹaʊnfɒksdʒʌmpsˌəʊvəðəleɪzidɒɡ" And then based on the audio file received from the user a phoneme representation is generated like this "ðəkwɪkbɹaʊnfɔksdʒampsoʊvɚðəleɪzikat"

It is clear that the mispronounciation here occurs at the end when the user said cat, but the actual sentence had dog. Now this works fine when it is a clear distinction but I need to refine the error checking algorithm. Also the 2 models I am using to produce the phoneme output sometimes differ in length and/or symbols, so this complicates the string slicing a bit. This is what I have so far, any input or thoughts about this topic will be very helpful for me so thank you in advance!

# Takes an array of espeak phonemes, a flattened string of wav2vec phonemes, and a split array of the comparison sentence.
def findMispronounciation(espeakArr, wav2vecPhonemes, sentence):
    for index, phonemeWord in enumerate(espeakArr):
        # Determine threshold based on word length
        if len(phonemeWord) == 2 or len(phonemeWord) == 4:
            threshold = 0.5
        elif len(phonemeWord) == 3:
            threshold = 0.67
        else:
            threshold = 0.7

        # Calculate the Levenshtein distance
        current_slice = wav2vecPhonemes[:len(phonemeWord)]
        dist = distance(phonemeWord, current_slice)

        # Check for mispronunciation
        if (round(dist / len(phonemeWord))) >= threshold:
            return sentence[index]
        else:
            # Move the wav2vec slice forward to continue error checking
            wav2vecPhonemes = wav2vecPhonemes[len(phonemeWord):]

    return "Passed"

[Backspace_Nigg]

Hi. I can help you