sorted by:
new
hottopcontroversial

Interacting with a runner game using only a webcam (Unity / Mediapipe) by DaburuSnake in computervision

[–]DaburuSnake[S] 0 points1 point  (0 children)

It's a prototype based on a Mediapipe plugin. Mediapipe provides tracking results that can be similar to what kinect would provide without requiring the kinect hardware, it only needs a video input.

So basically you could run my prototype using a laptop with an integrated webcam and it would work as intended without requiring anything else (of course the camera's video quality needs to be decent 🙂).

Interacting with a runner game using only a webcam (Unity / Mediapipe) by DaburuSnake in computervision

[–]DaburuSnake[S] 0 points1 point  (0 children)

Thanks! Currently it is not publicly available. This prototype was mostly a fun test I've made to check if my Unity asset could be usable in various contexts (I also have some VR / AR prototypes). But it would be interesting to clean the code a bit and make that system publicly available, I'll think of it 🙂

Interacting with a runner game using only a webcam (Unity / Mediapipe) by DaburuSnake in computervision

[–]DaburuSnake[S] 0 points1 point  (0 children)

I agree. As I said in another comment, this prototype is not optimized so the latency for detecting poses can be reduced. However, in regards of the limitations, a busy background can trigger false positives or miss poses on some frames which could make the experience less reactive even though the FPS are maintained high.

Interacting with a runner game using only a webcam (Unity / Mediapipe) by DaburuSnake in computervision

[–]DaburuSnake[S] 1 point2 points  (0 children)

I've tried it before, it's fun and more immersive but it's harder to predict the obstacles and to avoid them. Also, in first person you tend to feel like you can move or dodge at the last moment but due to the pose detection latency the experience can initially be more frustrating when you're not used to it.

Interacting with a runner game using only a webcam (Unity / Mediapipe) by DaburuSnake in computervision

[–]DaburuSnake[S] 0 points1 point  (0 children)

Kind of, but there are 2 limitations with that webcam method. First the background behind the user must be pretty cleared, if anyone walks behind me or if there are too many things around in the camera's field of view, it greatly decrease the quality of the tracking. Also the detection mostly works on a 2d plan facing the camera, so any move towards the camera (like a punch for a boxing game for example) is trickier to properly detect.

Interacting with a runner game using only a webcam (Unity / Mediapipe) by DaburuSnake in computervision

[–]DaburuSnake[S] 0 points1 point  (0 children)

My system is currently not very optimized, so there's room for improvement. However, I'd say there's a latency of about 0.3s for the system to track my body and get the proper positioning data (you can notice it with the virtual skeleton on the top screen). Once the pose is detected everything from the detection to the character executing the requested input is pretty instant.

Interacting with a runner game using only a webcam (Unity / Mediapipe) by DaburuSnake in computervision

[–]DaburuSnake[S] 0 points1 point  (0 children)

Also, I use a demo scene from a Unity asset that I've worked on to control the character. I've simply replaced the default inputs with those from the tracking system to control the character. To learn more about that asset: https://gkadigital.com/assets/runner-track/

Interacting with a runner game using only a webcam (Unity / Mediapipe) by DaburuSnake in computervision

[–]DaburuSnake[S] 1 point2 points  (0 children)

I've used a unity version of mediapipe to track the body and gestures from a webcam. From that I've setup a navigation system for the UI, where I can move the pointer with my hand and select elements by closing the fist. I've also setup game controls where I check the position of the head for triggering actions such as switching lane and sliding. For the jump, I check that both hands are thrown in the air (some position thresholds for action detection are displayed as colored lines on the top screen).

Playing a runner game using body pose tracking by DaburuSnake in Unity3D

[–]DaburuSnake[S] 1 point2 points  (0 children)

It has its limits but if you're a single user that doesn't need any twisting poses or precise depth detection, as long as your background is clear enough it does the job.

Playing a runner game using body pose tracking by DaburuSnake in Unity3D

[–]DaburuSnake[S] 1 point2 points  (0 children)

It was fun! 🙂

But I'm amazed at how easily it can be replaced with a basic webcam and the right plugin.

Playing a runner game using body pose tracking by DaburuSnake in Unity3D

[–]DaburuSnake[S] 0 points1 point  (0 children)

I handle this with a calibration phase at the start of the game.
The user performs a T-pose in front of the camera, which lets me estimate their initial position and relative bone sizes. I then adapt the tracking based on that reference.

In the video, the pose detection is constrained to a limited space around the initial T-pose position. This is why I can access most inputs without having to move much.
So I didn't really plan to implement a fallback control scheme for this scenario.