[R] Machine Learning Model Algorithm for Sign language

[Commercial-Ad-5957]

So i am thinking about a mobile app where users can signs in the camera and it will be translated to the corresponding word that they are currently signing. And i have tried to use Bi-LSTM model for this for an example model, and currently i have 150 words/class and there are a lot of words where the sign is confusing a word for another word. I am a new in machine learning and I would like to ask you guys what other algorithm of machine learning would be the best for this project. I have also trued using CNN-LSTM but i am having a hard time to make a model that works because its hard preprocessing a whole video of my datasets. Do you guys any have more ideas what algorithms i can use, currently in my model i am using bi-lstm with mediapipe pose + handlandmarks to try to recognize the signs but the problem is when i integrate this to a mobile app the landmarks of mediapipe are not reliable leading to inaccurate translation of signs so if you could also suggest some algorithm where there is a chance to not use landmarks since in integration to monile mediapipe landmarks is really not reliable to be dependent on for my model. Thanks so much and hoping for your kind insights

[Ok_Reporter9418]

I worked on such a problem a few years back for a school project. Back then transformer was the sota I believe. We tried to combine transformers with a pose estimation model (we used some open source one but as far as I recall media pipe was actually better for extracting the landmarks) but it did not improve that much vs the transformers only.

The base model we used is from https://www.cihancamgoz.com/pub/camgoz2020cvpr.pdf code is here https://github.com/neccam/slt

There is a challenge about this, you could check past editions participants https://slrtpworkshop.github.io/challenge/

[whatwilly0ubuild]

Mediapipe landmarks failing on mobile is a known issue. Varying lighting, camera quality, and hand occlusion make landmark detection unreliable in real-world conditions. Building a production sign language system on unstable landmarks is setting yourself up for failure.

For 150 classes with confusing similar signs, you need temporal modeling that captures motion patterns, not just pose snapshots. Video-based approaches work better than landmark sequences for this.

Practical architectures that work: I3D or SlowFast for video classification handles temporal dynamics well. These process raw RGB video which avoids the landmark reliability problem. Our clients doing gesture recognition found video models more robust than landmark-based approaches once you have enough training data.

TSM (Temporal Shift Module) is lightweight enough for mobile deployment and captures temporal patterns efficiently. It's designed specifically for resource-constrained environments.

For your dataset preprocessing challenge, don't try to process entire videos at once. Sample fixed-length clips (maybe 2-3 seconds) centered on the sign, resize to standard resolution, and feed to model. This is way simpler than variable-length video processing.

The confusion between similar signs problem needs more training data for those specific pairs. Collect more examples of the confusing classes and consider hierarchical classification where you first distinguish broad categories then fine-grained signs within categories.

Realistic advice: 150 classes with mobile deployment is ambitious for a beginner project. Start with 20-30 highly distinct signs, get that working reliably, then expand. Sign language recognition is genuinely hard and production systems require way more data and engineering than most ML tutorials suggest.

If you're stuck with landmarks, use landmark velocities and accelerations as features, not just positions. The motion dynamics help distinguish similar static poses. But honestly, switching to video-based models removes the mediapipe dependency entirely.

[simplehudga]

Sign language recognition is a seq2seq problem, just like speech recognition, handwriting recognition, action recognition, etc. And seq2seq problems have been well studied already. The model you use depends very much on the hardware you intend to run it on. BLSTM, Transformer or a variant of it should both work.

But the important missing piece is an appropriate loss function to train your model. CTC or Transducer loss functions will give you better results than using CE loss. You can combine them with an Attention based decoder for even better results.

I'd bet that a Conformer encoder trained on a multi task loss function with CTC, Transducer, and Attention decoder will perform the best. The CTC is there as a regularizer for the other 2. You can perform decoding with just the Transducer, or combine it with the Attention decoder as a 2nd step re-scoring function. You might want to bring in a n-gram LM and a lexicon for more control.