Generalization is a central challenge for the deployment of reinforcement
learning (RL) systems in the real world. In this paper, we show that the
sequential structure of the RL problem necessitates new approaches to
generalization beyond the well-studied techniques used in supervised learning.
While supervised learning methods can generalize effectively without explicitly
accounting for epistemic uncertainty, we show that, perhaps surprisingly, this
is not the case in RL. We show that generalization to unseen test conditions
from a limited number of training conditions induces implicit partial
observability, effectively turning even fully-observed MDPs into POMDPs.
Informed by this observation, we recast the problem of generalization in RL as
solving the induced partially observed Markov decision process, which we call
the epistemic POMDP. We demonstrate the failure modes of algorithms that do not
appropriately handle this partial observability, and suggest a simple
ensemble-based technique for approximately solving the partially observed
problem. Empirically, we demonstrate that our simple algorithm derived from the
epistemic POMDP achieves significant gains in generalization over current
methods on the Procgen benchmark suite.

本文研究了强化学习系统在现实世界中部署的中心挑战 —— 泛化，并展示了强化学习问题的时序结构需要新的泛化方法，同时介绍了一种新的部分可观察马尔可夫决策过程（POMDP）解决方案 —— 认知 POMDP，通过简单的基于集成的技术解决了部分可观测性问题，证明了所提出的算法对 Procgen 基准套件的表现相比现有方法有显著提高。

强化学习中泛化的困难之处：认知 POMDP 和隐式部分可观测性

Why Generalization in RL is Difficult: Epistemic POMDPs and Implicit  Partial Observability

Deep neural networks (DNNs) have achieved great success in solving a variety
of machine learning (ML) problems, especially in the domain of image
recognition. However, recent research showed that DNNs can be highly vulnerable
to adversarially generated instances, which look seemingly normal to human
observers, but completely confuse DNNs. These adversarial samples are crafted
by adding small perturbations to normal, benign images. Such perturbations,
while imperceptible to the human eye, are picked up by DNNs and cause them to
misclassify the manipulated instances with high confidence. In this work, we
explore and demonstrate how systematic JPEG compression can work as an
effective pre-processing step in the classification pipeline to counter
adversarial attacks and dramatically reduce their effects (e.g., Fast Gradient
Sign Method, DeepFool). An important component of JPEG compression is its
ability to remove high frequency signal components, inside square blocks of an
image. Such an operation is equivalent to selective blurring of the image,
helping remove additive perturbations. Further, we propose an ensemble-based
technique that can be constructed quickly from a given well-performing DNN, and
empirically show how such an ensemble that leverages JPEG compression can
protect a model from multiple types of adversarial attacks, without requiring
knowledge about the model.

使用 JPEG 压缩作为分类管道中的预处理步骤，可以在对抗性攻击中起到作用，通过移除高频信号成分从而消除加性扰动。我们还提出了一种基于集合的技术，该技术可以快速构建一个性能良好的 DNN 集成，通过利用 JPEG 压缩来保护模型免受多种类型的对抗性攻击，而不需要了解模型。