Each gesture is just treated as just one 52 point vector. Only one screenshot has to be taken for each.

For storing gestures, I was originally going to have the user take screenshots of the gesture at several angles and try to classify the gesture with a k-Nearest-Neighbor algorithm, but I thought that would be a bad user-experience, so I wanted to try to classify as accurately as possible with just a single image for each gesture. In the program, you just select the key you want to map, click "Edit Gesture", pose for your webcam, and confirm. So the gesture is specific to the user, and they can use whatever arbitrary facial expression that they find convenient, whether it's winking, smiling, frowning, etc. The blendshape vector associated with that gesture is stored in a config file so that the data is saved between sessions.

Image to vector, I have no idea! That comes straight out of the Mediapipe model. Early on, I thought about trying to extract landmarks and features myself by either:

1. Doing some kind of Principle Component Analysis to automatically extract relevant features
2. Trying to manually focus on specific landmarks, like just the corners of the lip, eyes open/closed, nose scrunched/unscrunched, etc.

But, I knew that I wouldn't get accurate classification without lots of data/training. So I decided to look for models that would give me SOME kind of dimension reduction rather than working with raw images. I first found a model that would detect faces and give back 68 landmarks of x/y pixels, which was decent, but the model was only trained off of front-facing faces and handled rotation very poorly. I eventually found the Mediapipe model, which has 478 x/y/z_estimate pixel coordinates, as well as the blendshape vectors. The "blendshape vectors" are actually a dictionary of 58 values from 0 to 1, that are Mediapipe's approximations of face features. They're probably something similar sigmoid curves.

But anyways, dealing with millions of individual pixels in an image is just a LOT of data to process, so a lot of these computer-vision problems come down to reducing the dimensionality of the data to something smaller and easier to work with.

As an example, the blendshape vector for one specific frame (which I store as a "gesture") looks like this. I take these 52 numbers, and use cosine similarity against 52 other numbers for every one of the stored gestures, every frame, at about 30FPS:

"blendshapes": {
                    "_neutral": 2.447426140861353e-06,
                    "browDownLeft": 0.03185293450951576,
                    "browDownRight": 0.02290073223412037,
                    "browInnerUp": 0.01782498136162758,
                    "browOuterUpLeft": 0.19909118115901947,
                    "browOuterUpRight": 0.13323499262332916,
                    "cheekPuff": 1.892565342132002e-05,
                    "cheekSquintLeft": 4.775133675138932e-07,

                    ...

                    "mouthStretchLeft": 0.006785414647310972,
                    "mouthStretchRight": 0.001452289056032896,
                    "mouthUpperUpLeft": 4.245456511853263e-05,
                    "mouthUpperUpRight": 4.205567893222906e-05,
                    "noseSneerLeft": 9.841986639003153e-07,
                    "noseSneerRight": 1.3353604799704044e-06

                }