PAC-Bayesian analysis is a frequentist framework for incorporating prior
knowledge into learning. It was inspired by Bayesian learning, which allows
sequential data processing and naturally turns posteriors from one processing
step into priors for the next. However, despite two and a half decades of
research, the ability to update priors sequentially without losing confidence
information along the way remained elusive for PAC-Bayes. While PAC-Bayes
allows construction of data-informed priors, the final confidence intervals
depend only on the number of points that were not used for the construction of
the prior, whereas confidence information in the prior, which is related to the
number of points used to construct the prior, is lost. This limits the
possibility and benefit of sequential prior updates, because the final bounds
depend only on the size of the final batch.
We present a novel and, in retrospect, surprisingly simple and powerful
PAC-Bayesian procedure that allows sequential prior updates with no information
loss. The procedure is based on a novel decomposition of the expected loss of
randomized classifiers. The decomposition rewrites the loss of the posterior as
an excess loss relative to a downscaled loss of the prior plus the downscaled
loss of the prior, which is bounded recursively. As a side result, we also
present a generalization of the split-kl and PAC-Bayes-split-kl inequalities to
discrete random variables, which we use for bounding the excess losses, and
which can be of independent interest. In empirical evaluation the new procedure
significantly outperforms state-of-the-art.

PAC-Bayesian analysis allows sequential prior updates with no information loss and has significant improvements in empirical evaluation.

递归 PAC-Bayes：一种在没有信息损失的情况下对顺序先验进行频率论更新的方法

Recursive PAC-Bayes: A Frequentist Approach to Sequential Prior Updates  with No Information Loss

Processing sequential data of variable length is a major challenge in a wide
range of applications, such as speech recognition, language modeling,
generative image modeling and machine translation. Here, we address this
challenge by proposing a novel recurrent neural network (RNN) architecture, the
Fast-Slow RNN (FS-RNN). The FS-RNN incorporates the strengths of both
multiscale RNNs and deep transition RNNs as it processes sequential data on
different timescales and learns complex transition functions from one time step
to the next. We evaluate the FS-RNN on two character level language modeling
data sets, Penn Treebank and Hutter Prize Wikipedia, where we improve state of
the art results to $1.19$ and $1.25$ bits-per-character (BPC), respectively. In
addition, an ensemble of two FS-RNNs achieves $1.20$ BPC on Hutter Prize
Wikipedia outperforming the best known compression algorithm with respect to
the BPC measure. We also present an empirical investigation of the learning and
network dynamics of the FS-RNN, which explains the improved performance
compared to other RNN architectures. Our approach is general as any kind of RNN
cell is a possible building block for the FS-RNN architecture, and thus can be
flexibly applied to different tasks.

提出了一种新的循环神经网络 Fast-Slow RNN，可以在不同的时间尺度上处理序列数据，并学习复杂的传递函数，实现了字符级语言建模数据集上的最佳效果。