Preference-based reinforcement learning (PbRL) has shown impressive
capabilities in training agents without reward engineering. However, a notable
limitation of PbRL is its dependency on substantial human feedback. This
dependency stems from the learning loop, which entails accurate reward learning
compounded with value/policy learning, necessitating a considerable number of
samples. To boost the learning loop, we propose SEER, an efficient PbRL method
that integrates label smoothing and policy regularization techniques. Label
smoothing reduces overfitting of the reward model by smoothing human preference
labels. Additionally, we bootstrap a conservative estimate $\widehat{Q}$ using
well-supported state-action pairs from the current replay memory to mitigate
overestimation bias and utilize it for policy learning regularization. Our
experimental results across a variety of complex tasks, both in online and
offline settings, demonstrate that our approach improves feedback efficiency,
outperforming state-of-the-art methods by a large margin. Ablation studies
further reveal that SEER achieves a more accurate Q-function compared to prior
work.

PbRL 方法 SEER 通过整合标签平滑和策略规则化技术，提高了反馈效率，取得了显著的性能优势。

通过对齐的经验估计实现高效的基于偏好的强化学习

Efficient Preference-based Reinforcement Learning via Aligned Experience  Estimation

Elucidating the reasoning process with structured explanations from question
to answer is fundamentally crucial, as it significantly enhances the
interpretability and trustworthiness of question-answering (QA) systems.
However, structured explanations demand models to perform intricate structured
reasoning, which poses great challenges. Most existing methods focus on
single-step reasoning through supervised learning, ignoring logical
dependencies between steps. Meanwhile, existing reinforcement learning
(RL)-based methods overlook the structured relationships, impeding RL's
potential in structured reasoning. In this paper, we propose SEER, a novel
method that maximizes a structure-based return to facilitate structured
reasoning and explanation. Our proposed structure-based return precisely
describes the hierarchical and branching structure inherent in structured
reasoning, effectively capturing the intricate relationships between states. We
also introduce a fine-grained reward function to meticulously delineate diverse
reasoning steps. Extensive experiments show that SEER significantly outperforms
state-of-the-art methods, achieving an absolute improvement of 6.9% over
RL-based methods on EntailmentBank, a 4.4% average improvement on STREET
benchmark, and exhibiting outstanding efficiency and cross-dataset
generalization performance.

通过提出 SEER 方法，我们在建立问答系统中，给出结构化解释，提高系统的可解释性和可靠性；实验证明，SEER 方法显著优于现有方法，在 EntailmentBank 上的绝对改进率达到了 6.9%，在 STREET 基准上平均提升了 4.4%，同时展现出卓越的效率和跨数据集的泛化性能。