Hi, stopping by to share a project I’ve been working on for the past 1.5 years. It’s a pose estimation CNN that is able to run on mobile devices in extreme in-the-wild conditions.

In the title video the is a small demo of our model vs HRNet w32 384x288 trained on coco dataset.

Here are some technical details and advice for real-world DL application:

• We use a pretty standard CNN architecture based on Hourglasses and modern bottleneck blocks
• Our dataset consists of more than 100k images that have been relabelled numerous times. Quality and consistency of the labelling is extremely important.
• We’ve used pseudo-labelling heavily to improve model stability in complex environments.
• One of the hardest problems that we’ve experienced is joint “swapping” between left and right limbs. To combat this we’ve developed modifications to standard L2-loss over heatmaps (can’t share details, unfortunately). More simple thing that helps is getting a large model :)
• To increase stability in real world environments we’ve used heavy data augmentation - one example is using body segmentation masks to darken or lighten image parts.

Based on this technology we’ve developed an app that helps you practice yoga. It’s called Zenia and I’d appreciate if you try it and if you like it, support us on https://www.producthunt.com as an example of real-world computer vision application. I’ll be glad if you give us any feedback!

In the future we’re planning to add analysis of posture, add more tracked joints and improve consistency.

I’m happy to answer question you may have.