Memory-augmented neural networks consisting of a neural controller and an
external memory have shown potentials in long-term sequential learning. Current
RAM-like memory models maintain memory accessing every timesteps, thus they do
not effectively leverage the short-term memory held in the controller. We
hypothesize that this scheme of writing is suboptimal in memory utilization and
introduces redundant computation. To validate our hypothesis, we derive a
theoretical bound on the amount of information stored in a RAM-like system and
formulate an optimization problem that maximizes the bound. The proposed
solution dubbed Uniform Writing is proved to be optimal under the assumption of
equal timestep contributions. To relax this assumption, we introduce
modifications to the original solution, resulting in a solution termed Cached
Uniform Writing. This method aims to balance between maximizing memorization
and forgetting via overwriting mechanisms. Through an extensive set of
experiments, we empirically demonstrate the advantages of our solutions over
other recurrent architectures, claiming the state-of-the-arts in various
sequential modeling tasks.

本论文研究了使用外部存储器的记忆增强神经网络，提出了一种更优的存储方案，得到了对多种序列建模任务的最优解。