Recurrent neural networks (RNNs) trained on compositional tasks can exhibit
functional modularity, in which neurons can be clustered by activity similarity
and participation in shared computational subtasks. Unlike brains, these RNNs
do not exhibit anatomical modularity, in which functional clustering is
correlated with strong recurrent coupling and spatial localization of
functional clusters. Contrasting with functional modularity, which can be
ephemerally dependent on the input, anatomically modular networks form a robust
substrate for solving the same subtasks in the future. To examine whether it is
possible to grow brain-like anatomical modularity, we apply a recent machine
learning method, brain-inspired modular training (BIMT), to a network being
trained to solve a set of compositional cognitive tasks. We find that
functional and anatomical clustering emerge together, such that functionally
similar neurons also become spatially localized and interconnected. Moreover,
compared to standard $L_1$ or no regularization settings, the model exhibits
superior performance by optimally balancing task performance and network
sparsity. In addition to achieving brain-like organization in RNNs, our
findings also suggest that BIMT holds promise for applications in neuromorphic
computing and enhancing the interpretability of neural network architectures.

RNNs trained on compositional tasks can exhibit functional and anatomical modularity when applying brain-inspired modular training, leading to improved performance and potential applications in neuromorphic computing and enhancing interpretability of neural network architectures.

不断成长的大脑：递归神经网络中解剖和功能模块化的协同发展

Growing Brains: Co-emergence of Anatomical and Functional Modularity in  Recurrent Neural Networks

Deep learning of artificial neural networks (ANNs) is creating highly
functional processes that are, unfortunately, nearly as hard to interpret as
their biological counterparts. Identification of functional modules in natural
brains plays an important role in cognitive and neuroscience alike, and can be
carried out using a wide range of technologies such as fMRI, EEG/ERP, MEG, or
calcium imaging. However, we do not have such robust methods at our disposal
when it comes to understanding functional modules in artificial neural
networks. Ideally, understanding which parts of an artificial neural network
perform what function might help us to address a number of vexing problems in
ANN research, such as catastrophic forgetting and overfitting. Furthermore,
revealing a network's modularity could improve our trust in them by making
these black boxes more transparent. Here, we introduce a new
information-theoretic concept that proves useful in understanding and analyzing
a network's functional modularity: the relay information $I_R$. The relay
information measures how much information groups of neurons that participate in
a particular function (modules) relay from inputs to outputs. Combined with a
greedy search algorithm, relay information can be used to identify
computational modules in neural networks. We also show that the functionality
of modules correlates with the amount of relay information they carry.

介绍了一种新信息理论概念 - 中继信息，该概念可用于理解和分析网络的功能模块化，并结合贪婪搜索算法用于识别神经网络中的计算模块，功能模块的功用与其所携带的中继信息量相关。