Identifying computational mechanisms for memorization and retrieval of data
is a long-standing problem at the intersection of machine learning and
neuroscience. Our main finding is that standard overparameterized deep neural
networks trained using standard optimization methods implement such a mechanism
for real-valued data. Empirically, we show that: (1) overparameterized
autoencoders store training samples as attractors, and thus, iterating the
learned map leads to sample recovery; (2) the same mechanism allows for
encoding sequences of examples, and serves as an even more efficient mechanism
for memory than autoencoding. Theoretically, we prove that when trained on a
single example, autoencoders store the example as an attractor. Lastly, by
treating a sequence encoder as a composition of maps, we prove that sequence
encoding provides a more efficient mechanism for memory than autoencoding.

本研究研究了计算机学习和神经科学交叉中记忆和检索数据的计算机机制，发现标准过度参数化的深度神经网络训练使用标准优化方法实现了一种用于实数数据的存储训练样本作为吸引子的 AutoEncoders 和序列编码提供了比 AutoEncoders 更有效的记忆机制。

过参数化神经网络实现联想记忆

Overparameterized Neural Networks Implement Associative Memory

Recurrent neural networks (RNNs) such as long short-term memory and gated
recurrent units are pivotal building blocks across a broad spectrum of sequence
modeling problems. This paper proposes a recurrently controlled recurrent
network (RCRN) for expressive and powerful sequence encoding. More concretely,
the key idea behind our approach is to learn the recurrent gating functions
using recurrent networks. Our architecture is split into two components - a
controller cell and a listener cell whereby the recurrent controller actively
influences the compositionality of the listener cell. We conduct extensive
experiments on a myriad of tasks in the NLP domain such as sentiment analysis
(SST, IMDb, Amazon reviews, etc.), question classification (TREC), entailment
classification (SNLI, SciTail), answer selection (WikiQA, TrecQA) and reading
comprehension (NarrativeQA). Across all 26 datasets, our results demonstrate
that RCRN not only consistently outperforms BiLSTMs but also stacked BiLSTMs,
suggesting that our controller architecture might be a suitable replacement for
the widely adopted stacked architecture.

该文提出了一种递归控制递归网络 (RCRN) 架构，利用递归网络学习递归门控函数，将其用于不同自然语言处理任务，结果表明 RCRN 在比双向 LSTM 和双向 LSTM 堆叠表现更好，具有代替双向 LSTM 堆叠的潜力。

循环控制循环网络

Recurrently Controlled Recurrent Networks

We propose MRU (Multi-Range Reasoning Units), a new fast compositional
encoder for machine comprehension (MC). Our proposed MRU encoders are
characterized by multi-ranged gating, executing a series of parameterized
contract-and-expand layers for learning gating vectors that benefit from long
and short-term dependencies. The aims of our approach are as follows: (1)
learning representations that are concurrently aware of long and short-term
context, (2) modeling relationships between intra-document blocks and (3) fast
and efficient sequence encoding. We show that our proposed encoder demonstrates
promising results both as a standalone encoder and as well as a complementary
building block. We conduct extensive experiments on three challenging MC
datasets, namely RACE, SearchQA and NarrativeQA, achieving highly competitive
performance on all. On the RACE benchmark, our model outperforms DFN (Dynamic
Fusion Networks) by 1.5%-6% without using any recurrent or convolution layers.
Similarly, we achieve competitive performance relative to AMANDA on the
SearchQA benchmark and BiDAF on the NarrativeQA benchmark without using any
LSTM/GRU layers. Finally, incorporating MRU encoders with standard BiLSTM
architectures further improves performance, achieving state-of-the-art results.

本文介绍了一种新型的机器理解编码器 MRU，具有多级门控、并行端到端的学习门控向量、适用于短期和长期依赖关系三个特点，为机器理解提供高效处理文档、优化模型性能、建模文档内部关系的方法。通过在三个具有挑战性的 MC 数据集上进行广泛的实验，我们证明了该编码器表现出了极具竞争力的性能表现与机器编码器和组件模块，是一种高性能状态的机器理解编码器。