Designing generalizable agents capable of adapting to diverse embodiments has
achieved significant attention in Reinforcement Learning (RL), which is
critical for deploying RL agents in various real-world applications. Previous
Cross-Embodiment RL approaches have focused on transferring knowledge across
embodiments within specific tasks. These methods often result in knowledge
tightly coupled with those tasks and fail to adequately capture the distinct
characteristics of different embodiments. To address this limitation, we
introduce the notion of Cross-Embodiment Unsupervised RL (CEURL), which
leverages unsupervised learning to enable agents to acquire embodiment-aware
and task-agnostic knowledge through online interactions within reward-free
environments. We formulate CEURL as a novel Controlled Embodiment Markov
Decision Process (CE-MDP) and systematically analyze CEURL's pre-training
objectives under CE-MDP. Based on these analyses, we develop a novel algorithm
Pre-trained Embodiment-Aware Control (PEAC) for handling CEURL, incorporating
an intrinsic reward function specifically designed for cross-embodiment
pre-training. PEAC not only provides an intuitive optimization strategy for
cross-embodiment pre-training but also can integrate flexibly with existing
unsupervised RL methods, facilitating cross-embodiment exploration and skill
discovery. Extensive experiments in both simulated (e.g., DMC and Robosuite)
and real-world environments (e.g., legged locomotion) demonstrate that PEAC
significantly improves adaptation performance and cross-embodiment
generalization, demonstrating its effectiveness in overcoming the unique
challenges of CEURL.

通过无监督学习以及预先训练的方式，提出了一种适用于不同体现形式的强化学习的控制策略，能够显著提高适应性能和跨体现形式的泛化能力。