Recent advances in legged locomotion have enabled quadrupeds to walk on
challenging terrains. However, bipedal robots are inherently more unstable and
hence it's harder to design walking controllers for them. In this work, we
leverage recent advances in rapid adaptation for locomotion control, and extend
them to work on bipedal robots. Similar to existing works, we start with a base
policy which produces actions while taking as input an estimated extrinsics
vector from an adaptation module. This extrinsics vector contains information
about the environment and enables the walking controller to rapidly adapt
online. However, the extrinsics estimator could be imperfect, which might lead
to poor performance of the base policy which expects a perfect estimator. In
this paper, we propose A-RMA (Adapting RMA), which additionally adapts the base
policy for the imperfect extrinsics estimator by finetuning it using model-free
RL. We demonstrate that A-RMA outperforms a number of RL-based baseline
controllers and model-based controllers in simulation, and show zero-shot
deployment of a single A-RMA policy to enable a bipedal robot, Cassie, to walk
in a variety of different scenarios in the real world beyond what it has seen
during training. Videos and results at this https URL

本文讨论如何利用快速适应的方法来控制走路姿态，并提出了 A-RMA（Adapting RMA）算法，它不仅可以适用于完美的外部估计器，还可以用于不完美的情况。实验表明，A-RMA 在仿真实验和现实世界中的表现均优于多种基准控制器。

为双足机器人适应快速运动适应性

Adapting Rapid Motor Adaptation for Bipedal Robots

In recent years, reinforcement learning (RL) has shown outstanding
performance for locomotion control of highly articulated robotic systems. Such
approaches typically involve tedious reward function tuning to achieve the
desired motion style. Imitation learning approaches such as adversarial motion
priors aim to reduce this problem by encouraging a pre-defined motion style. In
this work, we present an approach to augment the concept of adversarial motion
prior-based RL to allow for multiple, discretely switchable styles. We show
that multiple styles and skills can be learned simultaneously without notable
performance differences, even in combination with motion data-free skills. Our
approach is validated in several real-world experiments with a wheeled-legged
quadruped robot showing skills learned from existing RL controllers and
trajectory optimization, such as ducking and walking, and novel skills such as
switching between a quadrupedal and humanoid configuration. For the latter
skill, the robot is required to stand up, navigate on two wheels, and sit down.
Instead of tuning the sit-down motion, we verify that a reverse playback of the
stand-up movement helps the robot discover feasible sit-down behaviors and
avoids tedious reward function tuning.

本文提出了一种基于对抗性先验的多样式强化学习方法，通过引入多种风格的先验知识，成功实现了机器人在不同风格下的运动控制和技能学习，无需人工设计奖励函数。

通过多个对抗性运动先验在强化学习中进行高级技能训练

Advanced Skills through Multiple Adversarial Motion Priors in Reinforcement Learning

We propose to address quadrupedal locomotion tasks using Reinforcement
Learning (RL) with a Transformer-based model that learns to combine
proprioceptive information and high-dimensional depth sensor inputs. While
learning-based locomotion has made great advances using RL, most methods still
rely on domain randomization for training blind agents that generalize to
challenging terrains. Our key insight is that proprioceptive states only offer
contact measurements for immediate reaction, whereas an agent equipped with
visual sensory observations can learn to proactively maneuver environments with
obstacles and uneven terrain by anticipating changes in the environment many
steps ahead. In this paper, we introduce LocoTransformer, an end-to-end RL
method that leverages both proprioceptive states and visual observations for
locomotion control. We evaluate our method in challenging simulated
environments with different obstacles and uneven terrain. We transfer our
learned policy from simulation to a real robot by running it indoors and in the
wild with unseen obstacles and terrain. Our method not only significantly
improves over baselines, but also achieves far better generalization
performance, especially when transferred to the real robot. Our project page
with videos is at this https URL .

我们提出了一种使用基于 Transformer 的模型从配 proprioceptive 信息和高维度深度传感器输入中学习的强化学习（RL）来解决四足动物行走任务的方法。

使用跨模态转换器端到端学习视觉引导下的四足机器人运动

Learning Vision-Guided Quadrupedal Locomotion End-to-End with  Cross-Modal Transformers

Learning locomotion skills is a challenging problem. To generate realistic
and smooth locomotion, existing methods use motion capture, finite state
machines or morphology-specific knowledge to guide the motion generation
algorithms. Deep reinforcement learning (DRL) is a promising approach for the
automatic creation of locomotion control. Indeed, a standard benchmark for DRL
is to automatically create a running controller for a biped character from a
simple reward function. Although several different DRL algorithms can
successfully create a running controller, the resulting motions usually look
nothing like a real runner. This paper takes a minimalist learning approach to
the locomotion problem, without the use of motion examples, finite state
machines, or morphology-specific knowledge. We introduce two modifications to
the DRL approach that, when used together, produce locomotion behaviors that
are symmetric, low-energy, and much closer to that of a real person. First, we
introduce a new term to the loss function (not the reward function) that
encourages symmetric actions. Second, we introduce a new curriculum learning
method that provides modulated physical assistance to help the character with
left/right balance and forward movement. The algorithm automatically computes
appropriate assistance to the character and gradually relaxes this assistance,
so that eventually the character learns to move entirely without help. Because
our method does not make use of motion capture data, it can be applied to a
variety of character morphologies. We demonstrate locomotion controllers for
the lower half of a biped, a full humanoid, a quadruped, and a hexapod. Our
results show that learned policies are able to produce symmetric, low-energy
gaits. In addition, speed-appropriate gait patterns emerge without any guidance
from motion examples or contact planning.

该论文提出了一种新颖的基于深度强化学习（DRL）方法来训练机器人的步态控制算法，并引入了一种新的课程学习方法来提供适当的物理辅助，帮助机器人保持平衡和前进。