Go2 Extreme Parkour

Why this study

Reinforcement learning has taught legged robots to cross terrain that once took careful hand design. In Extreme Parkour with Legged Robots (Cheng et al., ICRA 2024), a low-cost quadruped with a single front-facing depth camera high-jumps onto obstacles twice its height and long-jumps across gaps twice its length, all with one learned policy.

Before designing my own study, I analyzed how this and two other well-known locomotion methods are built. The three set out to solve different problems. Rapid Motor Adaptation (RMA) (Kumar et al., RSS 2021) adapts a blind walking policy to changing physical conditions: in training, an encoder compresses privileged environment parameters such as friction and mass into a latent, and an adaptation module learns to estimate that latent from the robot's recent states and actions. Walk These Ways (Margolis & Agrawal, CoRL 2022) trains one blind policy to produce a family of gaits that can be tuned through its commands. Extreme Parkour adds terrain perception: its policy is first trained with height samples of the terrain and privileged information, then distilled into one that runs from the depth camera. For each, I traced what the networks observe, how those observations are represented, and how privileged information, which only the simulator can provide, is used in training.

My network analysis notes ↗RMA · Walk These Ways · Extreme Parkour

Comparing them sharpened the question I wanted to study: how terrain information should be represented, and what the Actor and the Critic should each observe. The robot can walk blind, read a raw height map, or read a compressed summary of it, and the Critic, which exists only during training, can be told things the Actor never sees. Rather than reproducing any one of these methods, I designed my own comparison on a Unitree Go2 in NVIDIA Isaac Lab: five Actor–Critic configurations that differ in how terrain and physics information reach each network, trained and evaluated under the same conditions. That first required a perceptive locomotion framework that could learn on hard terrain.

The learning framework

The policy controls the robot directly in joint space. Fifty times a second, it sets a target angle for each of the Go2's 12 joints, as an offset from a standing pose. It senses the ground through a height scan: 187 points on a 1.6 × 1.0 m grid around the robot, measured by ray casting in the simulator. The framework gives it heading-aware commands: a forward speed and a target heading, with the turning command set from how far the robot is from that heading, either sampled at random or held fixed. For the final network comparison, I fixed the command at 1 m/s forward with a heading of 0°, straight down each terrain strip, so that all five configurations met the same obstacles head-on under the same conditions.

Thousands of robots learn in parallel on a grid of terrain, and a terrain curriculum adjusts each robot's difficulty to its performance. A robot that gets far enough across its terrain in an episode moves up to a harder row; one that falls well short of where its command should have taken it moves down. Getting this to work on rough terrain took some debugging. At first, the robots' average level never rose past the first row. One cause was the reward for body height: it measured height in world coordinates, so a robot standing in a dip was penalized as if it were crouching. I changed it to measure height above the surrounding ground, read from the height scan, and reweighted the reward toward following the commanded speed until the robots climbed through the rows.

Earlier development: heading-aware commands & terrain curriculum ↗

The reward follows Extreme Parkour's as closely as I could make it, and took about 50 rounds of tuning. One bug was instructive. The energy term penalizes the work done by the motors, but when a leg swings hard, its torque and speed can point in opposite directions, so the work comes out negative and the penalty turns into a reward for flailing the legs. Counting only positive work fixed it. I also turned on collisions between the legs, which had been passing through each other. To keep the policy from depending on one exact robot, each robot's mass, center of mass, friction, and motor gains are randomized.

The course

Isaac Lab's default rough terrain turned out to be too easy for this comparison: in a first round, the configurations barely differed, in the training curves or in the videos. So I built a harder course of seven terrain types, modeled on the Extreme Parkour terrain: boxes, debris, hurdles, stairs, random rough ground, gaps, and parkour steps. It is a grid of 18 columns and 10 rows. Each column is one terrain type, laid out as a 23 m strip, and each row is harder than the one before. In the hardest row, the gaps are 80 cm wide, the parkour steps 45 cm high (almost the height of the robot's body), the hurdles 30 cm, and the stairs 23 cm.

Each configuration was trained with PPO on 4,096 robots in parallel for 20,000 iterations.

Five ways to give the networks the terrain

The policy is trained as two networks. The Actor decides how to move, and it is the part that would run on a real robot. The Critic exists only during training: it estimates how well the robot is doing so the Actor can learn, and it can be given information a real robot would not have.

Across the five configurations, the network inputs are organized into three observation groups:

• Robot state and command observations (52‑D): joint angles and velocities, the body's orientation and velocity, the command, the last action, and which feet are touching the ground. These 52 observations are provided to both the Actor and Critic in all five configurations.
• Height scan (187‑D).
• Physics parameters (29‑D): mass, center of mass, friction, and motor gains, the values randomized for each robot. Only the Critic ever sees these.

The scan can go in as it is, or first through a small encoder network (187 → 128 → 64 → 32) that compresses it into a 32‑D terrain latent. When both networks use encoded terrain observations (Abl 3.5 and 7.0), the Actor and Critic have separate scan encoders with independent weights. I compared five configurations. In the diagrams, blue marks the scan encoder and orange the physics encoder; select a diagram to enlarge it.

1. 1BlindAbl 1.0

   The Actor gets only the robot state and command observations; the Critic also gets the raw scan and the physics parameters.

   

2. 2Raw scanAbl 2.5

   The Actor gets the raw 187‑D scan too.

   

3. 3Encoded scan for bothAbl 3.5

   The Actor and the Critic each pass the scan through their own encoder and get a 32‑D terrain latent instead of the raw scan.

   

4. 4Encoded scan for the Actor onlyAbl 4.0

   The Actor gets the 32‑D latent; the Critic keeps the raw scan.

   

5. 5Configuration 3, plus encoded physicsAbl 7.0

   As in configuration 3, and a second, two-layer encoder compresses the Critic's 29‑D physics parameters into 20‑D.

   

Results

In the training curves, the blind and raw-scan policies finished clearly behind, with mean rewards of about 70 and 72, against 78 to 79 for the three that encode the scan. Those three ended within a point of each other, with configuration 3 slightly ahead in both reward and speed tracking at the last iteration. The runs on the hardest rows told them apart.

Mean reward

Velocity-tracking reward

• 1 Blind
• 2 Raw scan
• 3 Encoded scan for both
• 4 Encoded scan, Actor only
• 5 Configuration 3 + encoded physics

What the comparison showed

Seeing the terrain made anticipation possible. The blind policy still got across some gaps, but only by reacting: in the runs I compared, it found a gap when a front foot slipped into it and a step when it ran into it, then recovered with a large, sudden motion.

In these runs, the raw scan helped on steps but not on gaps. With the 187 raw heights, the Actor climbed the steps smoothly but fell into the first gap. With the 32‑D latent instead, the robot started its jump before the gap and landed with both front and rear legs. A compact latent may make features such as a gap's width or a step's height easier for the policy to use.

Encoding the scan for the Critic as well worked better in this setup. When only the Actor got the latent, the final reward was almost the same, but the robot picked up a habit of dragging its left feet on every terrain. One possible explanation is that the latent also gives the Critic steadier value estimates than 187 raw heights would, and so gives the Actor cleaner feedback to learn from.

Adding a physics encoder did not help. With the Critic's physics parameters encoded as well, the final reward was slightly lower and the gait got worse: the robot dragged two feet forward before crossing a gap and swung its rear-left leg uselessly on the steps. Because these parameters stay fixed for a whole episode, their encoded vector may have acted more like an identifier for each robot than like information about friction or mass that the Critic could generalize from.

My role

I conducted this study as a research intern at Sungkyunkwan University's Lab of Artificial Intelligence & Robotics (LAIR), working closely with Daegeol Ko, a senior researcher in the lab.

I developed the PPO-based perceptive locomotion framework in NVIDIA Isaac Lab, including the challenging terrain environments, heading-aware command sampling, curriculum learning, and reward functions.

The study grew out of our discussions about how different observations affect locomotion learning. Building on these discussions, I designed the five Actor–Critic observation and encoding configurations, developed the experimental methodology, and carried out the training and evaluation.

Throughout the project, we regularly discussed experimental setups and intermediate results, refining the study through successive iterations. I presented the study's progress and findings in a series of lab meetings. The implementation, network diagrams, and evaluation videos are available in my GitHub repository.

Next: CATCH

Although this project remained in simulation, it gave me valuable experience in learning-based locomotion, observation design, and experimental evaluation. I am now building on that experience through CATCH, a custom robot with a different embodiment that requires its own simulation, control development, and physical validation. Having studied policies for a fixed body on Go2 and now designing a new body with CATCH, I am drawn to a longer-term question: how a robot's body and its policy can be designed together.