Recurrent Neural Networks (RNNs) are renowned for their adeptness in modeling
temporal dependencies, a trait that has driven their widespread adoption for
sequential data processing. Nevertheless, vanilla RNNs are confronted with the
well-known issue of gradient vanishing and exploding, posing a significant
challenge for learning and establishing long-range dependencies. Additionally,
gated RNNs tend to be over-parameterized, resulting in poor network
generalization. To address these challenges, we propose a novel Delayed Memory
Unit (DMU) in this paper, wherein a delay line structure, coupled with delay
gates, is introduced to facilitate temporal interaction and temporal credit
assignment, so as to enhance the temporal modeling capabilities of vanilla
RNNs. Particularly, the DMU is designed to directly distribute the input
information to the optimal time instant in the future, rather than aggregating
and redistributing it over time through intricate network dynamics. Our
proposed DMU demonstrates superior temporal modeling capabilities across a
broad range of sequential modeling tasks, utilizing considerably fewer
parameters than other state-of-the-art gated RNN models in applications such as
speech recognition, radar gesture recognition, ECG waveform segmentation, and
permuted sequential image classification.

提出了一种新颖的延迟记忆单元（DMU），通过引入延迟线结构和延迟门，来增强普通 RNN 的时间建模能力，以解决梯度消失和梯度爆炸等问题，并在语音识别、雷达手势识别、心电波形分割和串联顺序图像分类等应用中展示出优越的时间建模能力。

延迟记忆单元：通过延迟门建模时间依赖

Delayed Memory Unit: Modelling Temporal Dependency Through Delay Gate

Large language models (LLMs) have significantly advanced the field of natural
language processing (NLP) through their extensive parameters and comprehensive
data utilization. However, existing LLMs lack a dedicated memory unit, limiting
their ability to explicitly store and retrieve knowledge for various tasks. In
this paper, we propose RET-LLM a novel framework that equips LLMs with a
general write-read memory unit, allowing them to extract, store, and recall
knowledge from the text as needed for task performance. Inspired by Davidsonian
semantics theory, we extract and save knowledge in the form of triplets. The
memory unit is designed to be scalable, aggregatable, updatable, and
interpretable. Through qualitative evaluations, we demonstrate the superiority
of our proposed framework over baseline approaches in question answering tasks.
Moreover, our framework exhibits robust performance in handling temporal-based
question answering tasks, showcasing its ability to effectively manage
time-dependent information.

提出了一种新的 RET-LLM 框架，它为 LLMs 提供了一个通用的写 - 读内存单元，使它们能够从文本中提取、存储和检索知识，以便进行任务执行。通过 Davidsonian 语义理论，以三元组的形式提取和保存知识，在问答任务中展现出比基线方法更卓越的性能。此外，该框架在处理基于时间的问题回答任务时表现出了强大的性能，展示了它有效处理时态信息的能力。