Cross-domain offline reinforcement learning leverages source domain data with
diverse transition dynamics to alleviate the data requirement for the target
domain. However, simply merging the data of two domains leads to performance
degradation due to the dynamics mismatch. Existing methods address this problem
by measuring the dynamics gap via domain classifiers while relying on the
assumptions of the transferability of paired domains. In this paper, we propose
a novel representation-based approach to measure the domain gap, where the
representation is learned through a contrastive objective by sampling
transitions from different domains. We show that such an objective recovers the
mutual-information gap of transition functions in two domains without suffering
from the unbounded issue of the dynamics gap in handling significantly
different domains. Based on the representations, we introduce a data filtering
algorithm that selectively shares transitions from the source domain according
to the contrastive score functions. Empirical results on various tasks
demonstrate that our method achieves superior performance, using only 10% of
the target data to achieve 89.2% of the performance on 100% target dataset with
state-of-the-art methods.

本文提出了一种基于表征的方法来测量领域差距，并通过对不同领域进行过渡采样来学习表征，从而解决合并两个领域数据导致性能下降的问题。通过基于表征的方法筛选数据，我们的实证结果表明，我们的方法仅使用目标数据的 10％就能以 89.2％的性能达到 100％目标数据集与最先进方法的性能。

跨领域离线强化学习中的数据过滤对比表示

Contrastive Representation for Data Filtering in Cross-Domain Offline  Reinforcement Learning

Generalizing policies across different domains with dynamics mismatch poses a
significant challenge in reinforcement learning. For example, a robot learns
the policy in a simulator, but when it is deployed in the real world, the
dynamics of the environment may be different. Given the source and target
domain with dynamics mismatch, we consider the online dynamics adaptation
problem, in which case the agent can access sufficient source domain data while
online interactions with the target domain are limited. Existing research has
attempted to solve the problem from the dynamics discrepancy perspective. In
this work, we reveal the limitations of these methods and explore the problem
from the value difference perspective via a novel insight on the value
consistency across domains. Specifically, we present the Value-Guided Data
Filtering (VGDF) algorithm, which selectively shares transitions from the
source domain based on the proximity of paired value targets across the two
domains. Empirical results on various environments with kinematic and
morphology shifts demonstrate that our method achieves superior performance
compared to prior approaches.

该研究提出了一种基于价值一致性的在线动态适应学习算法 —— 价值引导数据过滤算法（VGDF）。实验结果表明，与之前的方法相比，在不同领域的各种环境中，VGDF 算法在策略概括上表现更为优异。

价值引导数据过滤的跨域策略适应

Cross-Domain Policy Adaptation via Value-Guided Data Filtering

Reinforcement learning has shown great promise for synthesizing realistic
human behaviors by learning humanoid control policies from motion capture data.
However, it is still very challenging to reproduce sophisticated human skills
like ballet dance, or to stably imitate long-term human behaviors with complex
transitions. The main difficulty lies in the dynamics mismatch between the
humanoid model and real humans. That is, motions of real humans may not be
physically possible for the humanoid model. To overcome the dynamics mismatch,
we propose a novel approach, residual force control (RFC), that augments a
humanoid control policy by adding external residual forces into the action
space. During training, the RFC-based policy learns to apply residual forces to
the humanoid to compensate for the dynamics mismatch and better imitate the
reference motion. Experiments on a wide range of dynamic motions demonstrate
that our approach outperforms state-of-the-art methods in terms of convergence
speed and the quality of learned motions. Notably, we showcase a physics-based
virtual character empowered by RFC that can perform highly agile ballet dance
moves such as pirouette, arabesque and jet\'e. Furthermore, we propose a
dual-policy control framework, where a kinematic policy and an RFC-based policy
work in tandem to synthesize multi-modal infinite-horizon human motions without
any task guidance or user input. Our approach is the first humanoid control
method that successfully learns from a large-scale human motion dataset
(Human3.6M) and generates diverse long-term motions. Code and videos are
available at this https URL

该论文提出了使用 residual force control (RFC) 方法增强人型控制策略来更好地模仿长期人类动作，实现了学习源自大规模人体动作数据集的多模态无限地人类动作生成。