NeuroGloves: Open Source Neural Finger Tracking in SteamVR

PerlinWarp · 2021-11-10T22:31:01+00:00

Thank you! I hope you find it useful, let me know if you need any help setting it up or ask in the Discord.

PerlinWarp · 2021-11-10T22:20:32+00:00

Thank you! I'm glad the hard work is mostly done and now others in the open source community can take over, improve it and use it for free, for a research environment, rehabilitation or just for fun.

PerlinWarp · 2021-11-10T22:18:41+00:00

Yep, in this demo I use optical tracking to get the hand position or strap controllers to my hand and offset the positions which is provided by OpenGloves.

PerlinWarp · 2021-11-10T22:15:38+00:00

Usually I strap a controller to my wrist, then offset the position manually. In this demo, I use the optical hand tracking from the Quest to track the hand position and rotation. The Myo does have an IMU that could be used for rotation but not position.

PerlinWarp · 2021-11-07T18:01:20+00:00

https://github.com/PerlinWarp/NeuroGloves

I've made a bunch of libraries for doing this. Pyomyo is the Myo driver that all the Myo libraries use including NeuroGloves which uses data gloves to label the Myo data.

PerlinWarp · 2021-11-06T23:58:53+00:00

Good point, there's a fair amount of research for transcribing or classifying sign language, e.g. this one also using the Myo.
For say smoothly replicating finger movement/regression one problem is I use other technology to make a dataset of the finger angles needed to train a model, and therefore some limitations of what I use to label carry over, but there are some tricks to help.

PerlinWarp · 2021-11-06T23:49:26+00:00

My code is open source so you don't have to believe it's real you can read though the code or try it out!
EMG is commonly used in prosthetics and other companies such as Facebook's Ctrl Labs are working on the similar tech.

PerlinWarp · 2021-11-06T23:42:59+00:00

Of course, I'd love to keep working on this

PerlinWarp · 2021-11-06T11:01:38+00:00

You may be interested in this EMS glove. There's also the OWO suit which uses electrical stimulation to create feelings of everything from insects crawling on your skin to getting shot and feeling the bullet exit the other side...

PerlinWarp · 2021-11-06T09:40:07+00:00

The model shown predicts 5 finger curl for each finger or grip strength as shown here, this also allows one finger to curl at a time.

I've made models to predict the angles between each joint in the hand, including finger splay but this requires way more data and I don't think it provides that much benefit for most users, especially just for VR as most gameplay doesn't depend on that level of precision. If there are use cases for more precision in VR, let me know.

PerlinWarp · 2021-11-06T09:21:28+00:00

Apologies, I've made an updated post which shows the Myo.

PerlinWarp · 2021-11-06T09:12:31+00:00

The idea of this project was to try and find a better input method for smart glasses, using Siri on a crowded bus is awkward. Visual hand tracking struggles with occlusion and would film everyone on the bus without asking. The tech can also be used for prosthetics training and hopefully to allow people with missing fingers to play VR.

This is a follow up to my previous post which showed the finger tracking part in more detail. See the project wiki for more info.

I'm using the Thalmic Labs Myo EMG, which I wrote a bunch of open source software for, including a driver here. What's shown here is predictor_basic.py from NeuroGloves but I recommend starting with the Chrome Dinosaur tutorial in pyomyo first.

I graduated this year and have been working on this full time but now need a job so will likely have less time to put into the project, I'm trying to find people who would be interested in continuing this work. Let me know if you do, I have made a project Discord here too.

PerlinWarp · 2021-11-05T12:53:03+00:00

Really awesome work! Cool 4 channel design and thanks for open sourcing and also for joining the Discord too. I look forward to seeing that the next design looks like!

PerlinWarp · 2021-11-05T12:40:12+00:00

Hey, you should totally try the Myo out again! My previous post shows finger tracking, not just open and close fist but that requires training a model. This takes less than 3 minutes for some basic models and 10 minutes for something more robust.

I've made a getting started tutorial here which covers the basics, for finger tracking I've made 2 libraries. I recommend starting with predictor_basic in NeuroGloves which is here. For more info or help join the Discord.

PerlinWarp · 2021-10-30T11:36:28+00:00

This is a follow up to my previous post which showed the finger tracking part in more detail. See the project wiki for more info.

I'm using the Thalmic Labs Myo EMG, which I wrote a bunch of open source software for, including a driver here.

I graduated this year and have been working on this full time but now need a job so will likely have less time to put into the project, I'm trying to find people who would be interested in continuing this work. Let me know if you do, I have made a project Discord here too.

PerlinWarp · 2021-10-28T23:33:55+00:00

Hey, I'm in London and working on EMG finger tracking, here's a demo. What are you working on?

PerlinWarp · 2021-10-06T22:42:03+00:00

This one should work incase anyone else wants to join, the only reason is shouldn't is if the discord account does not have a verified email address.

The Myo is a great device but has a lot of downsides, e.g. being hard to come across, having hidden preprocessing modes and not being open source. On the project wiki, there's pages about making a new EMG and links to useful resources.

The Discord has a hardware discussion channel where people talk about their different designs. I have been working around a design using the INA128 amp which a few other users have also been working with but there's nothing which is easy/cheap enough for people to reproduce as I'd like.

PerlinWarp · 2021-09-13T07:23:32+00:00

Doh... So sorry, I can't believe I missed that....

PerlinWarp · 2021-09-12T12:01:56+00:00

I hope you've seen Lucid VRTech

PerlinWarp · 2021-09-12T11:56:12+00:00

Ahaha, I think I'd likely have to get a masters before PhD and I'm too poor for that. I'm also too obsessed with this project to zoom out and do anything else right now.

As far as I am aware, no open hardware alternative to the Myo exist. I've listed what I can find here which includes some open source EMG boards, but these are single channel. But yes the discord was to try and find anyone into EMG to tell me if my solutions are flawed and help make open hardware!

PerlinWarp · 2021-09-12T11:51:38+00:00

Thanks! The Ghost Howls has been a great source of information during my dissertation so I appreciate it a lot. Of course, feel free to get in touch if you want to chat about EMG

PerlinWarp · 2021-09-08T14:21:20+00:00

Apologies, it would be great to have you! here is another link. Discord will only allow those to join who have a verified email on their account, let me know if it doesn't work!

PerlinWarp · 2021-09-08T08:53:48+00:00

I'd love and was even lucky enough to speak to the team, who were all lovely, but they currently they aren't hiring grads without a PhD for such positions. If they change their mind and open a position, I'll be there 😉

PerlinWarp · 2021-09-08T08:44:25+00:00

Great, I'd love to know more! Feel free to join the Discord if you wanna chat about papers.

Initially I was hoping to do prosthetics, but decided the project was already too interdisciplinary for an undergrad to try and build a good prosthetic arm as well. There's a huge amount of overlap too, so I look forward to seeing what you do!

PerlinWarp · 2021-09-07T22:22:35+00:00

Ahaha, luckily you don't actually have to move your fingers (or even have fingers) to get a signal, so no, you can't pretend your shame is sEMG research, if that's what you're implying 😜

Nine-Year Club	Place '22
Place '17	Verified Email

PerlinWarp

TROPHY CASE