In offline reinforcement learning, the challenge of out-of-distribution (OOD)
is pronounced. To address this, existing methods often constrain the learned
policy through policy regularization. However, these methods often suffer from
the issue of unnecessary conservativeness, hampering policy improvement. This
occurs due to the indiscriminate use of all actions from the behavior policy
that generates the offline dataset as constraints. The problem becomes
particularly noticeable when the quality of the dataset is suboptimal. Thus, we
propose Adaptive Advantage-guided Policy Regularization (A2PR), obtaining
high-advantage actions from an augmented behavior policy combined with VAE to
guide the learned policy. A2PR can select high-advantage actions that differ
from those present in the dataset, while still effectively maintaining
conservatism from OOD actions. This is achieved by harnessing the VAE capacity
to generate samples matching the distribution of the data points. We
theoretically prove that the improvement of the behavior policy is guaranteed.
Besides, it effectively mitigates value overestimation with a bounded
performance gap. Empirically, we conduct a series of experiments on the D4RL
benchmark, where A2PR demonstrates state-of-the-art performance. Furthermore,
experimental results on additional suboptimal mixed datasets reveal that A2PR
exhibits superior performance. Code is available at
this https URL

通过自适应优势导向的策略规范化方法（A2PR），我们能够在离线强化学习中解决样本分布外问题，通过生成匹配数据点分布的样本，有效地选择高优势动作，并在保持一定保守性的同时改进行为策略，从而达到改进策略的理论保障，以及有效减轻价值函数高估的性能差距。

自适应优势引导的策略规范化用于离线强化学习

Adaptive Advantage-Guided Policy Regularization for Offline  Reinforcement Learning

We show that deep reinforcement learning can maintain its ability to learn
without resetting network parameters in settings where the number of gradient
updates greatly exceeds the number of environment samples. Under such large
update-to-data ratios, a recent study by Nikishin et al. (2022) suggested the
emergence of a primacy bias, in which agents overfit early interactions and
downplay later experience, impairing their ability to learn. In this work, we
dissect the phenomena underlying the primacy bias. We inspect the early stages
of training that ought to cause the failure to learn and find that a
fundamental challenge is a long-standing acquaintance: value overestimation.
Overinflated Q-values are found not only on out-of-distribution but also
in-distribution data and can be traced to unseen action prediction propelled by
optimizer momentum. We employ a simple unit-ball normalization that enables
learning under large update ratios, show its efficacy on the widely used
dm_control suite, and obtain strong performance on the challenging dog tasks,
competitive with model-based approaches. Our results question, in parts, the
prior explanation for sub-optimal learning due to overfitting on early data.

通过对深度强化学习的分析，我们发现在大量梯度更新次数远远超过环境样本数量的情况下，存在一种优先偏见现象，即代理程序过度依赖早期的交互并低估后期经验，从而影响其学习能力。我们发现这种现象的根本挑战在于价值过高估计，这不仅体现在样本外分布数据上，也表现在样本内分布数据上，并可追溯到由优化器动量推动的未知动作预测。我们采用了一种简单的单位球归一化方法，使得在大量梯度更新比例下学习变得可行，并在广泛使用的 dm_control 套件上取得了强大的性能，在具有挑战性的 dog 任务上与基于模型的方法相媲美。我们的结果对于早期数据过拟合导致学习不佳的先前解释提出了部分质疑。

高更新比例下的深度强化学习剖析：克服价值高估和发散

Dissecting Deep RL with High Update Ratios: Combatting Value  Overestimation and Divergence

We study worst-case guarantees on the expected return of fixed-dataset policy
optimization algorithms. Our core contribution is a unified conceptual and
mathematical framework for the study of algorithms in this regime. This
analysis reveals that for naive approaches, the possibility of erroneous value
overestimation leads to a difficult-to-satisfy requirement: in order to
guarantee that we select a policy which is near-optimal, we may need the
dataset to be informative of the value of every policy. To avoid this,
algorithms can follow the pessimism principle, which states that we should
choose the policy which acts optimally in the worst possible world. We show why
pessimistic algorithms can achieve good performance even when the dataset is
not informative of every policy, and derive families of algorithms which follow
this principle. These theoretical findings are validated by experiments on a
tabular gridworld, and deep learning experiments on four MinAtar environments.

本文研究了固定数据集策略优化算法的预期回报的最坏情况保证，并提出了一种统一的概念和数学框架来研究这个领域的算法，指出了朴素方法可能出现高估价值的情况。为了避免这种情况，可以使用悲观策略，该策略可以在数据集不完整的情况下实现良好表现，本文还给出了遵循悲观策略的算法家族，最后在一些实验中验证了这些理论结果。