The human brain possesses the extraordinary capability to contextualize the
information it receives from our environment. The entorhinal-hippocampal plays
a critical role in this function, as it is deeply engaged in memory processing
and constructing cognitive maps using place and grid cells. Comprehending and
leveraging this ability could significantly augment the field of artificial
intelligence. The multi-scale successor representation serves as a good model
for the functionality of place and grid cells and has already shown promise in
this role. Here, we introduce a model that employs successor representations
and neural networks, along with word embedding vectors, to construct a
cognitive map of three separate concepts. The network adeptly learns two
different scaled maps and situates new information in proximity to related
pre-existing representations. The dispersion of information across the
cognitive map varies according to its scale - either being heavily
concentrated, resulting in the formation of the three concepts, or spread
evenly throughout the map. We suggest that our model could potentially improve
current AI models by providing multi-modal context information to any input,
based on a similarity metric for the input and pre-existing knowledge
representations.

介绍了一种使用继任表示和神经网络的模型，以及单词嵌入向量，用于构建三个不同概念的认知图，从而根据输入提供多模态上下文信息，基于输入和预先存在的知识表示的相似性度量。

使用神经继承网络和词嵌入形成概念认知地图

Conceptual Cognitive Maps Formation with Neural Successor Networks and  Word Embeddings

Transformer-based models show their effectiveness across multiple domains and
tasks. The self-attention allows to combine information from all sequence
elements into context-aware representations. However, global and local
information has to be stored mostly in the same element-wise representations.
Moreover, the length of an input sequence is limited by quadratic computational
complexity of self-attention.
In this work, we propose and study a memory-augmented segment-level recurrent
Transformer (RMT). Memory allows to store and process local and global
information as well as to pass information between segments of the long
sequence with the help of recurrence.
We implement a memory mechanism with no changes to Transformer model by
adding special memory tokens to the input or output sequence. Then the model is
trained to control both memory operations and sequence representations
processing.
Results of experiments show that RMT performs on par with the Transformer-XL
on language modeling for smaller memory sizes and outperforms it for tasks that
require longer sequence processing. We show that adding memory tokens to Tr-XL
is able to improve its performance. This makes Recurrent Memory Transformer a
promising architecture for applications that require learning of long-term
dependencies and general purpose in memory processing, such as algorithmic
tasks and reasoning.

本文提出并研究了一种记忆增强的分段循环 Transformer 模型 (RMT)，通过通过加入特殊的记忆 tokens 实现，该模型能够存储和处理局部和全局信息，传递长序列片段之间的信息，具有学习长期依赖性和内存处理方面的潜力.

循环记忆变压器

Recurrent Memory Transformer

Mapping a truncated optimization method into a deep neural network, deep
unfolding network (DUN) has attracted growing attention in compressive sensing
(CS) due to its good interpretability and high performance. Each stage in DUNs
corresponds to one iteration in optimization. By understanding DUNs from the
perspective of the human brain's memory processing, we find there exists two
issues in existing DUNs. One is the information between every two adjacent
stages, which can be regarded as short-term memory, is usually lost seriously.
The other is no explicit mechanism to ensure that the previous stages affect
the current stage, which means memory is easily forgotten. To solve these
issues, in this paper, a novel DUN with persistent memory for CS is proposed,
dubbed Memory-Augmented Deep Unfolding Network (MADUN). We design a
memory-augmented proximal mapping module (MAPMM) by combining two types of
memory augmentation mechanisms, namely High-throughput Short-term Memory (HSM)
and Cross-stage Long-term Memory (CLM). HSM is exploited to allow DUNs to
transmit multi-channel short-term memory, which greatly reduces information
loss between adjacent stages. CLM is utilized to develop the dependency of deep
information across cascading stages, which greatly enhances network
representation capability. Extensive CS experiments on natural and MR images
show that with the strong ability to maintain and balance information our MADUN
outperforms existing state-of-the-art methods by a large margin. The source
code is available at this https URL

提出一种新颖的具有持久性记忆的深度展开网络（MADUN）用于压缩感知问题，利用高吞吐量短期记忆和跨阶段长期记忆机制来保持和平衡信息，从而增强网络表示能力和性能，超过现有的最先进方法。