Although reinforcement learning has found widespread use in dense reward
settings, training autonomous agents with sparse rewards remains challenging.
To address this difficulty, prior work has shown promising results when using
not only task-specific demonstrations but also task-agnostic albeit somewhat
related demonstrations. In most cases, the available demonstrations are
distilled into an implicit prior, commonly represented via a single deep net.
Explicit priors in the form of a database that can be queried have also been
shown to lead to encouraging results. To better benefit from available
demonstrations, we develop a method to Combine Explicit and Implicit Priors
(CEIP). CEIP exploits multiple implicit priors in the form of normalizing flows
in parallel to form a single complex prior. Moreover, CEIP uses an effective
explicit retrieval and push-forward mechanism to condition the implicit priors.
In three challenging environments, we find the proposed CEIP method to improve
upon sophisticated state-of-the-art techniques.

本文提出一种名为 CEIP 的方法，通过使用多个并行的流来形成一个单一复杂的先验分布，并使用有效的显式检索和推进机制来改善强化学习中通过利用内在和外在先验信息来解决稀疏奖励的问题。

CEIP: 结合显式和隐式先验知识的强化学习

CEIP: Combining Explicit and Implicit Priors for Reinforcement Learning with Demonstrations

Prior probability models are a fundamental component of many image processing
problems, but density estimation is notoriously difficult for high-dimensional
signals such as photographic images. Deep neural networks have provided
state-of-the-art solutions for problems such as denoising, which implicitly
rely on a prior probability model of natural images. Here, we develop a robust
and general methodology for making use of this implicit prior. We rely on a
statistical result due to Miyasawa (1961), who showed that the least-squares
solution for removing additive Gaussian noise can be written directly in terms
of the gradient of the log of the noisy signal density. We use this fact to
develop a stochastic coarse-to-fine gradient ascent procedure for drawing
high-probability samples from the implicit prior embedded within a CNN trained
to perform blind (i.e., with unknown noise level) least-squares denoising. A
generalization of this algorithm to constrained sampling provides a method for
using the implicit prior to solve any linear inverse problem, with no
additional training. We demonstrate this general form of transfer learning in
multiple applications, using the same algorithm to produce state-of-the-art
levels of unsupervised performance for deblurring, super-resolution,
inpainting, and compressive sensing.

本文提出了一种利用深度神经网络模型中的隐式先验进行线性反问题求解的方法，借助宫澤统计学理论和随机梯度上升算法，在去噪、去模糊、插值和压缩感知等方面取得了最先进的非监督性能水平。