Convolutional neural nets (CNNs) have demonstrated remarkable performance in
recent history. Such approaches tend to work in a unidirectional bottom-up
feed-forward fashion. However, practical experience and biological evidence
tells us that feedback plays a crucial role, particularly for detailed spatial
understanding tasks. This work explores bidirectional architectures that also
reason with top-down feedback: neural units are influenced by both lower and
higher-level units.
We do so by treating units as rectified latent variables in a quadratic
energy function, which can be seen as a hierarchical Rectified Gaussian model
(RGs). We show that RGs can be optimized with a quadratic program (QP), that
can in turn be optimized with a recurrent neural network (with rectified linear
units). This allows RGs to be trained with GPU-optimized gradient descent. From
a theoretical perspective, RGs help establish a connection between CNNs and
hierarchical probabilistic models. From a practical perspective, RGs are well
suited for detailed spatial tasks that can benefit from top-down reasoning. We
illustrate them on the challenging task of keypoint localization under
occlusions, where local bottom-up evidence may be misleading. We demonstrate
state-of-the-art results on challenging benchmarks.

本研究探索了双向的神经网络结构和基于顶部反馈的推理方式，通过将神经元作为具有二次能量函数的整流潜变量，它可以被视为分层整流高尔基模型，并展示了其在具有挑战性的关键点本地化任务上的领先表现。

使用层次化修正高斯进行自下而上和自上而下的推理

Bottom-Up and Top-Down Reasoning with Hierarchical Rectified Gaussians

Countless learning tasks require dealing with sequential data. Image
captioning, speech synthesis, and music generation all require that a model
produce outputs that are sequences. In other domains, such as time series
prediction, video analysis, and musical information retrieval, a model must
learn from inputs that are sequences. Interactive tasks, such as translating
natural language, engaging in dialogue, and controlling a robot, often demand
both capabilities. Recurrent neural networks (RNNs) are connectionist models
that capture the dynamics of sequences via cycles in the network of nodes.
Unlike standard feedforward neural networks, recurrent networks retain a state
that can represent information from an arbitrarily long context window.
Although recurrent neural networks have traditionally been difficult to train,
and often contain millions of parameters, recent advances in network
architectures, optimization techniques, and parallel computation have enabled
successful large-scale learning with them. In recent years, systems based on
long short-term memory (LSTM) and bidirectional (BRNN) architectures have
demonstrated ground-breaking performance on tasks as varied as image
captioning, language translation, and handwriting recognition. In this survey,
we review and synthesize the research that over the past three decades first
yielded and then made practical these powerful learning models. When
appropriate, we reconcile conflicting notation and nomenclature. Our goal is to
provide a self-contained explication of the state of the art together with a
historical perspective and references to primary research.

本文对于近三十年来产生和实践了重要的循环神经网络（RNN），LSTM 和 BRNN 等模型的研究进行综述，旨在提供一个自成体系的最前沿阐述和历史视角，并引用了相关研究文献。