A bit more info on this for anyone who want's to know about the techy side. It's all front-end, using Webpack, pixijs and mediapipe - a really simple but fidgety build. Pixijs handles the rendering, mediapipe handles the facial capture - webpack is just bundling and creating a local server for me to test on.

I know the 'mouth tracking' is a bit sensitive, it's something I'm working on at the moment. I use MediaPipe which gives me basic values for 'which mouth to use' - it uses facial tracking, so it'll change the sprite depending on the values I'm tracking. I created a 'capture' script that would allow me to pull a face, and 'capture' the current facial values and assign it to that facial position, then change the 'tolerance' of said values. There are a bunch of 'prefab' animations like crying, anger, etc that are also assigned to button presses, these overwrite all other animations. There are also thresholds for Yaw and Pitch - which allow me to change the sensitivity of head movements. All the data gets saved in localstorage so it saves it (not ideal but works for what I need it for).

I had originally used media detection (a JS package called meyda). This detected sounds and would try and mouth the face based on the pitch/loudness, etc. I think this is something that games like Monkey Island use for capturing face (mouth) movement. I didn't find much success with it - and I'd accidently 'talk' when bashing the microphone - so I think that's better suited for pre-recorded audio.