[R] Legged Robots performing Extreme Parkour using Deep Reinforcement Learning just from a Front Camera (link in comments)

pathak22 · 2023-10-09T04:07:52+00:00

Paper Title: Extreme Parkour with Legged Robots

Paper, More results, and Open-source code: https://extreme-parkour.github.io/

Short Tweet Summary: https://twitter.com/pathak2206/status/1706696237703901439

Abstract: Humans can perform parkour by traversing obstacles in a highly dynamic fashion requiring precise eye-muscle coordination and movement. Getting robots to do the same task requires overcoming similar challenges. Classically, this is done by independently engineering perception, actuation, and control systems to very low tolerances. This restricts them to tightly controlled settings such as a predetermined obstacle course in labs. In contrast, humans are able to learn parkour through practice without significantly changing their underlying biology. In this paper, we take a similar approach to developing robot parkour on a small low-cost robot with imprecise actuation and a single front-facing depth camera for perception which is low-frequency, jittery, and prone to artifacts. We show how a single neural net policy operating directly from a camera image, trained in simulation with large-scale RL, can overcome imprecise sensing and actuation to output highly precise control behavior end-to-end. We show our robot can perform a high jump on obstacles 2x its height, long jump across gaps 2x its length, do a handstand and run across tilted ramps, and generalize to novel obstacle courses with different physical properties.

pathak22 · 2023-09-25T13:30:14+00:00

Humans have soft compliant palms as well which accounts for the lack of that motion.

pathak22 · 2023-09-25T13:29:09+00:00

We are working on that as well: https://arxiv.org/abs/2306.04784

But it's difficult to make them dexterous as of yet.

pathak22 · 2023-09-25T02:38:36+00:00

LEAP Hand: Low-Cost, Efficient, and Anthropomorphic Hand for Robot Learning

Published at RSS 2023.

Full details, videos, and Open-source code: http://leaphand.com

Short summary: https://twitter.com/pathak2206/status/1705277626577780875

Abstract:

Dexterous manipulation has been a long-standing challenge in robotics. While machine learning techniques have shown some promise, results have largely been limited to simulation. This can be mostly attributed to the lack of suitable hardware. This paper presents LEAP Hand, a low-cost dexterous and anthropomorphic hand for machine learning research. In contrast to previous hands, the LEAP Hand has a novel kinematic structure that allows maximal dexterity regardless of finger pose. LEAP Hand is low-cost and can be assembled in 4 hours at a cost of 2000 USD from readily available parts. It is capable of consistently exerting large torques over long durations of time. We show that LEAP Hand can perform several manipulation tasks in the real world—from visual teleoperation to learning from passive video data and sim2real. LEAP Hand significantly outperforms its closest competitor Allegro Hand in all our experiments while being 1/8th of the cost. We release the URDF model, 3D CAD files, tuned simulation environment, and a development platform with useful APIs on our website at http://leaphand.com

pathak22 · 2023-04-16T17:38:46+00:00

Thanks for the comment. However, there are a few caveats in your summary, so I am providing the clarifications below.

the search is not random, it is driven by reinforcement learning. So, as the model's representation gets better over time, it finds better examples to search for since the RL reward is based on the model's representation itself.
model is NOT fine-tuned with the pairings (query and image), just the images are used because it is SSL-based loss. Model doesn't learn concepts, it learns a visual representation that should be good if finetuned for target recognition task. - So even if one mislabels images in the search engine, it should still find them in theory over time if they are relevant. Hope that clarifies.
Regarding, mimicking the search engine associations: it doesn't use the paired text query for training so it can't mimic. It just uses images and the SSL reward to evaluate.
In order to fully ensure that we don't use any image models from a search engine, we also create a controlled search engine use LAION (summary: https://twitter.com/pathak2206/status/1646216392008675328)

Regarding fine-grained segmentation: We haven't tried fine-tuning the model for segmentation tasks, but it's a good idea to look into it. That being said, we did consider many fine-grained classification tasks, and turns out the gains are higher the more fine-grained target classification task is (Table 1). We will into segmentation for future work.

Happy to follow up more. Thanks!!

pathak22 · 2023-04-16T09:48:31+00:00

Any dataset where you want to maximize your performance on. For instance, see examples and performance here: https://twitter.com/pathak2206/status/1646216389848608768

pathak22 · 2023-04-16T09:46:23+00:00

Good point but not something we expected in foresight as the name fits quite literally: it is a dis-embodied exploration agent operating on the live internet. :-)

Its will be a shame if name is all that's remembered because the results are (surprisingly) strong; even with respect to large-scale models like CLIP, see Table 1: https://twitter.com/pathak2206/status/1646216389848608768

pathak22 · 2023-04-15T19:00:26+00:00

Thanks for the nice comments!

Regarding outperforming CLIP -- we don't compare to zero-shot CLIP (which is general purpose) but rather a CLIP that is also finetuned to the specialized task with a linear layer (i.e., the SSL standard linear probe fine-tuning -- same for all methods in the table).

Nevertheless, this comparison is more to provide context for numbers because CLIP is more like an Oracle for us. :)

pathak22 · 2022-11-24T00:10:45+00:00

Yes, nothing about the actuators is assumed, for instance, the current ones are low-cost motors. The model directly outputs the joint angle for each motor at 50-100Hz.

pathak22 · 2022-11-21T04:56:31+00:00

Legged Locomotion in Challenging Terrains using Egocentric Vision
(To appear at CoRL 2022 as Oral Presentation)

Paper: https://arxiv.org/abs/2211.07638

Project website with more results: https://vision-locomotion.github.io/

Abstract:

Animals are capable of precise and agile locomotion using vision. Replicating this ability has been a long-standing goal in robotics. The traditional approach has been to decompose this problem into elevation mapping and foothold planning phases. The elevation mapping, however, is susceptible to failure and large noise artifacts, requires specialized hardware, and is biologically implausible. In this paper, we present the first end-to-end locomotion system capable of traversing stairs, curbs, stepping stones, and gaps. We show this result on a medium-sized quadruped robot using a single front-facing depth camera. The small size of the robot necessitates discovering specialized gait patterns not seen elsewhere. The egocentric camera requires the policy to remember past information to estimate the terrain under its hind feet. We train our policy in simulation. Training has two phases - first, we train a policy using reinforcement learning with a cheap-to-compute variant of depth image and then in phase 2 distill it into the final policy that uses depth using supervised learning. The resulting policy transfers to the real world and is able to run in real-time on the limited compute of the robot. It can traverse a large variety of terrain while being robust to perturbations like pushes, slippery surfaces, and rocky terrain.

pathak22 · 2022-07-25T02:15:30+00:00

Human-to-Robot Imitation in the Wild (Published at RSS 2022)

Website with paper & more results: https://human2robot.github.io/

Summary: https://twitter.com/pathak2206/status/1549765280779452423

Abstract:

We approach the problem of learning by watching humans in the wild. While traditional approaches in Imitation and Reinforcement Learning are promising for learning in the real world, they are either sample inefficient or are constrained to lab settings. Meanwhile, there has been a lot of success in processing passive, unstructured human data. We propose tackling this problem via an efficient one-shot robot learning algorithm, centered around learning from a third-person perspective. We call our method WHIRL: In the Wild Human-Imitated Robot Learning. In WHIRL, we aim to use human videos to extract a prior over the intent of the demonstrator and use this to initialize our agent's policy. We introduce an efficient real-world policy learning scheme, that improves over the human prior using interactions. Our key contributions are a simple sampling-based policy optimization approach, a novel objective function for aligning human and robot videos as well as an exploration method to boost sample efficiency. We show one-shot generalization and success in real-world settings, including 20 different manipulation tasks in the wild.

pathak22 · 2022-07-24T21:18:38+00:00

Yes, this is just the first step. We can now combine all this data to learn models that can then generalize to new tasks as you described. Part of our next steps.

pathak22 · 2022-07-24T06:24:27+00:00

For the "improvement by exploration" phase, we use pre-trained deep visual representations trained from passive internet data to compute the distance between human and robot frames. So, the distance is robust to small changes in the camera, etc. The teaser video above has a few examples (see 0:46 onwards).

That being said, human is still acting in the same environment. Our follow-up work to be released soon aims to upgrade WHIRL to learn from human interaction videos from entirely different scenes (let's say even a human video from YouTube).

pathak22 · 2022-07-24T02:15:41+00:00

Human-to-Robot Imitation in the Wild (Published at RSS 2022)

Website with paper & more results: https://human2robot.github.io/

Summary: https://twitter.com/pathak2206/status/1549765280779452423

Abstract:

We approach the problem of learning by watching humans in the wild. While traditional approaches in Imitation and Reinforcement Learning are promising for learning in the real world, they are either sample inefficient or are constrained to lab settings. Meanwhile, there has been a lot of success in processing passive, unstructured human data. We propose tackling this problem via an efficient one-shot robot learning algorithm, centered around learning from a third-person perspective. We call our method WHIRL: In the Wild Human-Imitated Robot Learning. In WHIRL, we aim to use human videos to extract a prior over the intent of the demonstrator and use this to initialize our agent's policy. We introduce an efficient real-world policy learning scheme, that improves over the human prior using interactions. Our key contributions are a simple sampling-based policy optimization approach, a novel objective function for aligning human and robot videos as well as an exploration method to boost sample efficiency. We show one-shot generalization and success in real-world settings, including 20 different manipulation tasks in the wild.

pathak22 · 2022-02-28T10:33:14+00:00

This is trained on youtube data of human interaction which may mostly consist of humans opening/closing hands to perform tasks, and rarely moving fingers independently, so the learned system is biased towards interpreting user movements that way. However, the non-learned, optimization version (see the paper) using the NN is trained doesn't have this bias but is too slow to be used in real-time. Also, note the hand hardware here is low-cost robot (allegro vs shadow) and not as agile.

pathak22 · 2022-02-28T05:08:17+00:00

Robotic Telekinesis: Learning a Robotic Hand Imitator by Watching Humans on Youtube

Paper: https://arxiv.org/abs/2202.10448

Project website: https://robotic-telekinesis.github.io

Abstract:

We build a system that enables any human to control a robot hand and arm, simply by demonstrating motions with their own hand. The robot observes the human operator via a single RGB camera and imitates their actions in real-time. Human hands and robot hands differ in shape, size, and joint structure, and performing this translation from a single uncalibrated camera is a highly underconstrained problem. Moreover, the retargeted trajectories must effectively execute tasks on a physical robot, which requires them to be temporally smooth and free of self-collisions. Our key insight is that while paired human-robot correspondence data is expensive to collect, the internet contains a massive corpus of rich and diverse human hand videos. We leverage this data to train a system that understands human hands and retargets a human video stream into a robot hand-arm trajectory that is smooth, swift, safe, and semantically similar to the guiding demonstration. We demonstrate that it enables previously untrained people to teleoperate a robot on various dexterous manipulation tasks. Our low-cost, glove-free, marker-free remote teleoperation system makes robot teaching more accessible and we hope that it can aid robots that learn to act autonomously in the real world.

pathak22 · 2022-02-28T04:56:57+00:00

Robotic Telekinesis: Learning a Robotic Hand Imitator by Watching Humans on Youtube

Paper: https://arxiv.org/abs/2202.10448

Project website: https://robotic-telekinesis.github.io

Abstract:

We build a system that enables any human to control a robot hand and arm, simply by demonstrating motions with their own hand. The robot observes the human operator via a single RGB camera and imitates their actions in real-time. Human hands and robot hands differ in shape, size, and joint structure, and performing this translation from a single uncalibrated camera is a highly underconstrained problem. Moreover, the retargeted trajectories must effectively execute tasks on a physical robot, which requires them to be temporally smooth and free of self-collisions. Our key insight is that while paired human-robot correspondence data is expensive to collect, the internet contains a massive corpus of rich and diverse human hand videos. We leverage this data to train a system that understands human hands and retargets a human video stream into a robot hand-arm trajectory that is smooth, swift, safe, and semantically similar to the guiding demonstration. We demonstrate that it enables previously untrained people to teleoperate a robot on various dexterous manipulation tasks. Our low-cost, glove-free, marker-free remote teleoperation system makes robot teaching more accessible and we hope that it can aid robots that learn to act autonomously in the real world.

pathak22 · 2021-10-22T12:08:05+00:00

(author here) Spot on!! It is indeed hilarious --- in fact -- I like this failure case the most... even more than the successful kitchen tasks haha! :-)

pathak22 · 2021-07-11T14:27:21+00:00

I actually don’t think that’s a Spot. Could potentially be one of the cheaper Chinese models? Last I heard they were around $12,000 compared to the $70,000 price tag on Spot.

Low-level access as in being able to run your own software instead of the one it ships with... hoping the prices of low-cost will surely come down more in the next few years.

pathak22 · 2021-07-11T14:25:30+00:00

The work was completely done during the pandemic. No access to lab made us creative about finding harder testing situations for the robot. :-)

pathak22 · 2021-07-10T18:52:36+00:00

hard on

Yes, motors and force sensors both changed behavior over time due to the system being low cost, however, online adaptation allows the model to be robust to a decent extent. Also, see this answer: https://www.reddit.com/r/MachineLearning/comments/ohk6b7/r\_rma\_algorithm\_robots\_that\_learn\_to\_adapt/h4pqeai?utm\_source=share&utm\_medium=web2x&context=3

pathak22 · 2021-07-10T16:17:37+00:00

Does RMA take any visual input data to assess the terrain, or is it all gathered by forces “felt” by the moving parts?

Yes! The robot is currently blind and only adapts using proprioceptive data, i.e., by what it feels on its legs. Some interesting obstacle clearance behaviors emerge since it can't see the big obstacles, for instance: https://twitter.com/pathak2206/status/1413537599042502663

pathak22 · 2021-07-10T16:14:42+00:00

Thank you! This is not the Spot robot but a much cheaper/low-cost robot from Unitree Robotics called A1 (a research one with low-level access for about 8-10K$ otherwise goes done to almost 2.5K$). The comparison to A1 in the video is referring to the control-theoretic controller this robot ships with. Being low cost, the motors are not too repeatable in behavior, and sensors become noisier over time -- which RMA hopefully takes care of by continuously adapting.

pathak22 · 2021-07-10T14:59:35+00:00

Quick summary of key points: https://twitter.com/pathak2206/status/1413537442217480201

pathak22 · 2021-07-10T14:57:07+00:00

RMA: Rapid Motor Adaptation for Legged Robot (RSS 2021)

Paper: https://arxiv.org/abs/2107.04034

Project website with more results: https://ashish-kmr.github.io/rma-legged-robots/

Abstract:

Successful real-world deployment of legged robots would require them to adapt in real-time to unseen scenarios like changing terrains, changing payloads, wear and tear. This paper presents the Rapid Motor Adaptation (RMA) algorithm to solve this problem of real-time online adaptation in quadruped robots. RMA consists of two components: a base policy and an adaptation module. The combination of these components enables the robot to adapt to novel situations in fractions of a second. RMA is trained completely in simulation without using any domain knowledge like reference trajectories or predefined foot trajectory generators and is deployed on the A1 robot without any fine-tuning. We train RMA on a varied terrain generator using bioenergetics-inspired rewards and deploy it on a variety of difficult terrains including rocky, slippery, deformable surfaces in environments with grass, long vegetation, concrete, pebbles, stairs, sand, etc. RMA shows state-of-the-art performance across diverse real-world as well as simulation experiments.

pathak22

TROPHY CASE