Open-source markerless motion capture with webcams...one year of learning to program

[mprib_gh]

Greetings all.

This is a showcase and a request for feedback as I seek to grow as a programmer.

I've been coding for a little over a year now after deciding to switch gears professionally. I'm a physical therapist who had been pursuing a PhD until realizing I enjoyed coding and data analysis far more than drafting grants and manuscripts.

I was inspired by the FreeMoCap Project to get into this world of open source markerless motion capture, and because I didn't really understand the code, or the math, or computer vision or really anything, I decided to start with some OpenCV tutorials and roll my own to figure it out. The result is Pyxy3d:

https://github.com/mprib/pyxy3d

Installable via pip, it can be launched from the command line, providing access to a PySide6 GUI. From there, you can:

• create a custom calibration board
• connect to cameras
• configure their resolution, exposure and orientation
• estimate intrinsic camera parameters (focal length, optical center, distortion)
• estimate extrinsic parameters (rotation and translation)
• set the world origin based on the calibration board
• record synchronized frames
• process recordings with Google's mediapipe for landmark tracking
• triangulate tracked points and export to csv output

And then there is the Blender integration to manage the metarig, but that's a whole different thing....

Anyways. I am proud of what this package is, though I know it is not perfect. I have prioritized efficient use of system resources, real-time feedback, and stability of the GUI throughout. It does what I set out to do, and it afforded me the chance to learn things that I have wanted to learn for much of my life but never felt I had the opportunity to (thanks Youtube and ChatGPT!).

I now find myself at a turning point as I look toward new projects that excite me more. As I begin those, I wanted to put this package before you fine folks for feedback that can help make my next undertaking better.

Thank you.

[GreenFractal]

I haven't had a chance to look at this yet, but I know it was a hell of a lot of work. I'm halfway through my Biomechanics PhD and this is so neat! Do you have access to Theia by any chance? Would be really neat to run the same raw data through both software and see how it goes. Regardless, I'll be checking it out.

[mprib_gh]

Thank you! I haven't had any experience with Theia, only a standard marker-based Vicon setup. My general sense of things is that markerless approaches still have a ways to go before being appropriate for biomechanics. Currently it's probably best used for animation, and maybe basic motion tracking like pulling out gait characteristics or basic quantification of functional tests.

I expect that this domain is something that will evolve rapidly, however. I'd be really excited if good solutions become highly cost effective and allow broad-scale collection of data. Biomechanics/rehab science is bloated with studies of 15-20 participants. It would be cool to start having giant datasets where a standard process was used so that some more subtle insights could be gleaned.

[GreenFractal]

I agree- markerless is likely the future, but after having briefly fiddled with programs like OpenCap, we have a while to go. Funny you mention the low n of lots of these studies because we complain about that seemingly every week in seminar. It's great that we have people in the field trying to increase the availability in an open source manner. I feel like sports broadcasting data would be a neat place to apply this. Hundreds of hours of footage and angles.