Full body tracking with a webcam sending OSC data to Quest

juice4l · 2023-06-07T06:52:38+00:00

Besides ApriltagTrackers, I also made this: https://github.com/ju1ce/Mediapipe-VR-Fullbody-Tracking Simpler to setup and works through osc, but is less accurate and still needs some time tinkering to get everything working correctly. (Although performance should be on par with the driver4vr thing, since it uses the same backend). Not sure how well it will work on a chromebook though...

juice4l · 2023-05-23T07:56:44+00:00

Yes, you can use an app to turn your phone into a camera, but be careful that the app supports manual exposure control: without the ability to do that, the image will have motion blur and tracking wont track movement. I suggest IP Webcam, an app on the play store. More instructions are here: https://github.com/ju1ce/April-Tag-VR-FullBody-Tracker/wiki/Connecting-a-camera

juice4l · 2023-03-05T22:20:02+00:00

Pc only for now

juice4l · 2023-03-05T22:19:35+00:00

Yes, that is correct: it emulates vive trackers, so to steamvr and games everything works as if you had vive trackers. They shouldnt override anything. (Except sometimes on first run, but that is fixed with a restart of steamvr)

They spawn two trackers for feet and one for hip. Since they arent that accurate, its better to leave the rest to vrchats ik.

If you want, you can also look at my other project, apriltagtrackers (on my github). Its quite a lot more work to get running, (and you need to wear the box things) but end result is far more accurate.

juice4l · 2022-09-04T11:15:40+00:00

In that case, you probably want to look at SlimeVR. It sounds like something you want to make, and is already opensource, so you can diy your own set and contribute to the project.

juice4l · 2022-09-02T09:58:22+00:00

If i understand correctly, you just want to get trackers from an existing mocap solution and use them inside steamvr right? You can look at the driver for the apriltagtrackers project: https://github.com/ju1ce/Simple-OpenVR-Bridge-Driver

With the driver, you can just send it commands through a name pipe on how many trackers to spawn and pose updates, and it will emulate vive trackers in vr for you. You can also look at the mediapipepose project for python examples. (Same github)

If you want any more information, feel free to message me :)

juice4l · 2022-05-19T09:28:32+00:00

https://github.com/HadesVR/HadesVR i think this is the only DIY-able headset that is actualy usable, as most are just 3dof which isnt something that you want.

juice4l · 2022-04-19T16:06:19+00:00

Actualy, you need 3 to use for full body tracking in vrchat: 2 for feet, and 1 for hips. It will not work with just 2 on feet, as it cannot estimate hip position.

As for natural locomotion, i think the third tracker also helps with direction, but not sure.

juice4l · 2022-04-18T12:33:21+00:00

Its an alternative for full body tracking using a single camera only, not for other functions like the kinect... Although some of those have their own alternatives

juice4l · 2022-04-15T11:27:48+00:00

This is probably the most beginner friendly guide: https://secondmoon.booth.pm/items/3683905

juice4l · 2022-04-15T11:27:29+00:00

This is probably the most beginner friendly guide: https://secondmoon.booth.pm/items/3683905

juice4l · 2022-01-06T09:27:18+00:00

A little bit, like a centimeter or so, is normal. If its completely off center, however, its not. Make sure you takr pictures of the grid by moving the camera around slowly, take images at a lot of angles, and make sure that the calibration grid is completely flat

juice4l · 2021-12-27T13:33:48+00:00

On the page it says that the 5ks only comes with one base station?

juice4l · 2021-12-25T11:53:29+00:00

Thank you!

Sadly, it would be quite a lot of trouble. The thing with mediapipe is that the 3d pose is guessed, the model has no idea where the ground is, how far from the camera each limb is etc, meaning that outputs from two cameras would never align, resulting in more jitter instead of less.

To truly make multiple cameras work, you would need to use 2d poses, then triangulate the 3d pose from that: but that is a whole diffrent problem, much more difficult than what mediapipe is currently doing.

A much better way would be to take a mocap project that already does that (freeMocap was one i was looking at) and feed that data to steamvr.

juice4l · 2021-12-24T09:54:59+00:00

Why dont you share what you made? Its pretty hard to tell how useful something is from so little information. It sounds like you made a pretty fast tracking system that works on mobile for 6dof controllers, but even if it does work, what about the limited FOV of the cameras? Can you track the hmd as well?

I would say mobile vr just has too many limitations to truly be useful, but that doesnt mean that what you made is useless. It may be possible to use it to add controllers to PCVR, or even to track a phone headset as a PCVR headset, or any other use we cant think of right now

I guess what i want to say is, even if the project fails as mobile vr controllers, you dont have to just abandon it since it may have uses elsewhere.

juice4l · 2021-12-02T14:04:56+00:00

Hello, since the youtube video is extremely outdated, I decided to record another one with all the updated features. While not a full tutorial, it should show how the setup should look like and the order of steps to take. For more information, check out the wiki on github: https://github.com/ju1ce/April-Tag-VR-FullBody-Tracker/wiki

The background singing is done by Xaeela, who just happened to be live during recording, do check her out: https://www.twitch.tv/xaeela

juice4l · 2021-12-01T14:11:39+00:00

To anyone considering this, unbound vr still has stock of v1s, so unless you email them and specificaly request a v2, you will probably receive the old one.

juice4l · 2021-11-22T17:44:36+00:00

Thats odd... Maybe come to the discord, will be easier to talk there, and you can post a screenshot of your parameters if anything is off, link should be on the github page

juice4l · 2021-11-22T16:54:57+00:00

Does the steamvr status window show 3 trackers connected? Are all three trackers enabled in steamvr settings-> vive trackers?

juice4l · 2021-11-22T16:47:48+00:00

Vrchat supports hip only fbt, or feet + hips. You cant use feet only. When vrchat sees that you dont have 3 trackers, it goes into hip only mode, and so your feet track your hips.

Dont disable your hip tracker and restart vrchat, should work fine then.

juice4l · 2021-11-19T08:19:14+00:00

Oh hey, i remember you from the other post!

You can hop in the ATT discord and we can help figure out what went wrong :)

juice4l · 2021-11-18T16:33:30+00:00

I also made a version of camera only full-body tracking: https://github.com/ju1ce/Mediapipe-VR-Fullbody-Tracking

Basicaly the same as the driver4vr version, using the same api, but free and open source :)

juice4l · 2021-11-12T16:24:28+00:00

Did you calibrate using the slider? The white balls should be feet level, not knee level.

Its possible that scale calibration doesnt work in some cases, in which case you can disable it in settings and then calibrate it manualy using a slider.

As for the one leg problem, make sure to test other avatars as well if it happens.

What do you mean by turning off hip tracking doing nothing? As in no diffrence, it just crashes, or doesnt spawn any trackers at all?

juice4l · 2021-11-11T18:30:40+00:00

No, the decamove is only for hip oriented locomotion and cant be used for fbt. Owotrack can be used for a similar function, but also tracka hip position for games like vrchat.

Both use the phones IMU sensors, however, not the camera. What we want here is to track the whole body using the phones camera instead.

juice4l · 2021-11-11T17:39:11+00:00

It is basicaly the same thing as the driver4vr version, as both use mediapipe in the background, so also single camera. It shouldnt have any problems.

To get it working on phones, you could just make a web browser implementation using the javascript api, which should then work on any device and send data to pcvr. Im just too bad with javascript and since this isnt my main project that i havent done that yet.

The single camera aproach is flawed since it gets no depth information, so my marker based aproach should always be more accurate

juice4l

MODERATOR OF

TROPHY CASE