Cross-domain offline reinforcement learning leverages source domain data with
diverse transition dynamics to alleviate the data requirement for the target
domain. However, simply merging the data of two domains leads to performance
degradation due to the dynamics mismatch. Existing methods address this problem
by measuring the dynamics gap via domain classifiers while relying on the
assumptions of the transferability of paired domains. In this paper, we propose
a novel representation-based approach to measure the domain gap, where the
representation is learned through a contrastive objective by sampling
transitions from different domains. We show that such an objective recovers the
mutual-information gap of transition functions in two domains without suffering
from the unbounded issue of the dynamics gap in handling significantly
different domains. Based on the representations, we introduce a data filtering
algorithm that selectively shares transitions from the source domain according
to the contrastive score functions. Empirical results on various tasks
demonstrate that our method achieves superior performance, using only 10% of
the target data to achieve 89.2% of the performance on 100% target dataset with
state-of-the-art methods.

本文提出了一种基于表征的方法来测量领域差距，并通过对不同领域进行过渡采样来学习表征，从而解决合并两个领域数据导致性能下降的问题。通过基于表征的方法筛选数据，我们的实证结果表明，我们的方法仅使用目标数据的 10％就能以 89.2％的性能达到 100％目标数据集与最先进方法的性能。

跨领域离线强化学习中的数据过滤对比表示

Contrastive Representation for Data Filtering in Cross-Domain Offline  Reinforcement Learning

Recent research shows that pre-trained language models (PLMs) suffer from
"prompt bias" in factual knowledge extraction, i.e., prompts tend to introduce
biases toward specific labels. However, the extent and impact of prompt bias
within the model remain underexplored. In response, this paper quantifies the
bias with various types of prompts and assesses their impact on different
benchmarks. We show that: 1) all prompts in the experiments exhibit
non-negligible bias, with gradient-based prompts like AutoPrompt and OptiPrompt
displaying significantly higher levels of bias; 2) prompt bias can amplify
benchmark accuracy unreasonably by overfitting the test datasets, especially on
imbalanced datasets like LAMA. Based on these findings, we propose a
representation-based approach to mitigate the prompt bias during inference
time. Specifically, we first estimate the biased representation using
prompt-only querying, and then remove it from the model's internal
representations to generate the debiased representations, which are used to
produce the final debiased outputs. Experiments across various prompts, PLMs,
and benchmarks show that our approach can not only correct the overfitted
performance caused by prompt bias, but also significantly improve the prompt
retrieval capability (up to 10% absolute performance gain). Our findings shed
new light on the underlying predicting mechanisms of prompt-based queries in
PLMs. Hopefully, our plug-and-play approach can be a golden standard to
strengthen PLMs toward reliable knowledge bases. Code and data are released in
this https URL

该研究量化了各种类型提示的偏差，并评估了其对不同基准测试的影响。通过提出一种基于表示的方法来缓解提示偏差，研究发现该方法不仅可以纠正由提示偏差引起的过拟合性能，还可以显著提高提示检索能力。