In this paper we analyze the gate activation signals inside the gated
recurrent neural networks, and find the temporal structure of such signals is
highly correlated with the phoneme boundaries. This correlation is further
verified by a set of experiments for phoneme segmentation, in which better
results compared to standard approaches were obtained.

本文分析了门控循环神经网络中的门激活信号，并发现这些信号的时间结构与音素边界高度相关。通过一系列音素分割实验进一步验证了这种相关性，与标准方法相比，取得了更好的结果。

门控循环神经网络的门控激活信号分析及其与音素边界的相关性

Gate Activation Signal Analysis for Gated Recurrent Neural Networks and  Its Correlation with Phoneme Boundaries

Sophisticated gated recurrent neural network architectures like LSTMs and
GRUs have been shown to be highly effective in a myriad of applications. We
develop an un-gated unit, the statistical recurrent unit (SRU), that is able to
learn long term dependencies in data by only keeping moving averages of
statistics. The SRU's architecture is simple, un-gated, and contains a
comparable number of parameters to LSTMs; yet, SRUs perform favorably to more
sophisticated LSTM and GRU alternatives, often outperforming one or both in
various tasks. We show the efficacy of SRUs as compared to LSTMs and GRUs in an
unbiased manner by optimizing respective architectures' hyperparameters in a
Bayesian optimization scheme for both synthetic and real-world tasks.

通过 Bayesian optimization 方案优化模型超参数，我们展示了统计循环单元 (SRU) 相比于门控循环神经网络 (LSTM) 和门控 GRU 的高效性。

统计复发单元

The Statistical Recurrent Unit

We present a model of visually-grounded language learning based on stacked
gated recurrent neural networks which learns to predict visual features given
an image description in the form of a sequence of phonemes. The learning task
resembles that faced by human language learners who need to discover both
structure and meaning from noisy and ambiguous data across modalities. We show
that our model indeed learns to predict features of the visual context given
phonetically transcribed image descriptions, and show that it represents
linguistic information in a hierarchy of levels: lower layers in the stack are
comparatively more sensitive to form, whereas higher layers are more sensitive
to meaning.

基于堆叠门控循环神经网络的视觉语言学习模型，可以通过音素序列描述的图像描述，学习预测图像的视觉特征，并以层级结构表示语言信息。

从音素到图像：一种循环神经模型中的表征层次结构，用于视觉语言学习

From phonemes to images: levels of representation in a recurrent neural  model of visually-grounded language learning

Whereas deep neural networks were first mostly used for classification tasks,
they are rapidly expanding in the realm of structured output problems, where
the observed target is composed of multiple random variables that have a rich
joint distribution, given the input. We focus in this paper on the case where
the input also has a rich structure and the input and output structures are
somehow related. We describe systems that learn to attend to different places
in the input, for each element of the output, for a variety of tasks: machine
translation, image caption generation, video clip description and speech
recognition. All these systems are based on a shared set of building blocks:
gated recurrent neural networks and convolutional neural networks, along with
trained attention mechanisms. We report on experimental results with these
systems, showing impressively good performance and the advantage of the
attention mechanism.

本文介绍了使用深度神经网络中的递归神经网络和卷积神经网络等措施来实现注意力机制，以解决多元随机变量富含联合分布的结构化输出问题。通过典型任务如机器翻译、图像字幕生成、视频片段描述和语音识别实验，系统展示了很好的性能。