Reinforcement Learning (RL) has achieved remarkable success in
safety-critical areas, but it can be weakened by adversarial attacks. Recent
studies have introduced "smoothed policies" in order to enhance its robustness.
Yet, it is still challenging to establish a provable guarantee to certify the
bound of its total reward. Prior methods relied primarily on computing bounds
using Lipschitz continuity or calculating the probability of cumulative reward
above specific thresholds. However, these techniques are only suited for
continuous perturbations on the RL agent's observations and are restricted to
perturbations bounded by the l_2-norm. To address these limitations, this paper
proposes a general black-box certification method capable of directly
certifying the cumulative reward of the smoothed policy under various
$l_p$-norm bounded perturbations. Furthermore, we extend our methodology to
certify perturbations on action spaces. Our approach leverages f-divergence to
measure the distinction between the original distribution and the perturbed
distribution, subsequently determining the certification bound by solving a
convex optimisation problem. We provide a comprehensive theoretical analysis
and run sufficient experiments in multiple environments. Our results show that
our method not only improves the certified lower bound of mean cumulative
reward but also demonstrates better efficiency than state-of-the-art
techniques.

本文提出了一种通用的黑盒认证方法，能够在各种 $l_p$ 范数边界扰动下直接认证平滑策略的累积奖励，并扩展了在动作空间上认证扰动的方法。我们利用 f - 分布度量原始分布与扰动分布之间的差异，并通过求解凸优化问题确定认证边界。理论分析和实验结果表明，我们的方法不仅提高了平均累积奖励的认证下界，而且比最先进的技术更高效。

策略平滑强化学习的奖励认证

Reward Certification for Policy Smoothed Reinforcement Learning

We present a new certification method for image and point cloud segmentation
based on randomized smoothing. The method leverages a novel scalable algorithm
for prediction and certification that correctly accounts for multiple testing,
necessary for ensuring statistical guarantees. The key to our approach is
reliance on established multiple-testing correction mechanisms as well as the
ability to abstain from classifying single pixels or points while still
robustly segmenting the overall input. Our experimental evaluation on synthetic
data and challenging datasets, such as Pascal Context, Cityscapes, and
ShapeNet, shows that our algorithm can achieve, for the first time, competitive
accuracy and certification guarantees on real-world segmentation tasks. We
provide an implementation at this https URL

提出了一种基于随机平滑的图像和点云分割的新认证方法，利用多重测试机制确保统计保证，并能够对单一像素或点进行分类性弃权，实现对整个输入的鲁棒分割，实验证明算法在真实世界的分割任务中具有具有竞争力的精度和认证保证。