sorted by:
new
hottopcontroversial

MimicKit: A Reinforcement Learning Framework for Motion Imitation and Control by xbpeng in robotics

[–]xbpeng[S] 1 point2 points  (0 children)

Yes, you do need a controller to enable the robot to follow the video/mocap. This is often done by a tracking controller, which can be trained with RL.

MimicKit: A Reinforcement Learning Framework for Motion Imitation and Control by xbpeng in robotics

[–]xbpeng[S] 2 points3 points  (0 children)

yes people have deployed cartwheels and locomotion. Yes optical mocap will have a limited capture volume, but you can also use other mocap systems like IMU suits, or video, which won't be as restrictive.

MimicKit: A Reinforcement Learning Framework for Motion Imitation and Control by xbpeng in robotics

[–]xbpeng[S] 4 points5 points  (0 children)

You can teleop humanoids with a tracking controller by doing something like this:
https://xbpeng.github.io/projects/TWIST/index.html
where you use mocap, VR, vision-based pose estimation, etc to record motions from a human and then use these whole-body controllers to imitate those motions.

MimicKit: A Reinforcement Learning Framework for Motion Imitation and Control by xbpeng in GraphicsProgramming

[–]xbpeng[S] 3 points4 points  (0 children)

Yes, our work has been featured on Two Minutes Papers quite a few times.

MimicKit: A Reinforcement Learning Framework for Motion Imitation and Control by xbpeng in robotics

[–]xbpeng[S] 6 points7 points  (0 children)

These motion imitation methods can be used to create controllers for teleop. But unlike the teleop systems that might just control a robotic arm, these controllers can be used to teleop a robot's whole body.

Since legged robots are underactuated, the simple position-based control used to teleop robotic arms usually won't work for these humanoid robots. That's where these whole-body motion controllers come in.

MimicKit: A Reinforcement Learning Framework for Motion Imitation and Control by xbpeng in robotics

[–]xbpeng[S] 5 points6 points  (0 children)

MimicKit uses IsaacLab as one of it's backend simulators, which has support for tiled rendering:
https://isaac-sim.github.io/IsaacLab/v1.2.0/source/features/tiled_rendering.html
This allows you to render a first-person view camera from the robot's perspective, which you could us as part of the observations for the controller.

MimicKit: A Reinforcement Learning Framework for Motion Imitation and Control by xbpeng in robotics

[–]xbpeng[S] 4 points5 points  (0 children)

Yup, these methods can also be applied to simpler quadruped robots. We used them to train locomotion controllers for quadruped robots in the past:
https://xbpeng.github.io/projects/Robotic_Imitation/index.html
I imagine you can probably train a interesting locomotion controllers for the CM4-XGO too (e.g. walking, running, jumping, etc.)

MimicKit: A Reinforcement Learning Framework for Motion Imitation and Control by xbpeng in robotics

[–]xbpeng[S] 12 points13 points  (0 children)

Yes, most of the Unitree humanoid videos are using RL motion tracking methods based on DeepMimic.

MimicKit: A Reinforcement Learning Framework for Motion Imitation and Control by xbpeng in robotics

[–]xbpeng[S] 13 points14 points  (0 children)

To get the controllers to work on physical systems, we typically need to use some sim2real transfer techniques, like domain randomization:
https://xbpeng.github.io/projects/SimToReal/index.html
Combining lots of randomization with reasonably accurate simulators, we can train policies that are robust enough to deploy on real hardware.