Large language models based on self-attention mechanisms have achieved
astonishing performances not only in natural language itself, but also in a
variety of tasks of different nature. However, regarding processing language,
our human brain may not operate using the same principle. Then, a debate is
established on the connection between brain computation and artificial
self-supervision adopted in large language models. One of most influential
hypothesis in brain computation is the predictive coding framework, which
proposes to minimize the prediction error by local learning. However, the role
of predictive coding and the associated credit assignment in language
processing remains unknown. Here, we propose a mean-field learning model within
the predictive coding framework, assuming that the synaptic weight of each
connection follows a spike and slab distribution, and only the distribution is
trained. This meta predictive learning is successfully validated on classifying
handwritten digits where pixels are input to the network in sequence, and on
the toy and real language corpus. Our model reveals that most of the
connections become deterministic after learning, while the output connections
have a higher level of variability. The performance of the resulting network
ensemble changes continuously with data load, further improving with more
training data, in analogy with the emergent behavior of large language models.
Therefore, our model provides a starting point to investigate the physics and
biology correspondences of the language processing and the unexpected general
intelligence.

基于自注意力机制的大型语言模型在语言处理和自然语言本身中取得了惊人的成绩，并且在不同性质的各种任务中也取得了成绩。本文在预测编码框架内提出了一个平均场学习模型，成功地验证了其在分类手写数字和玩具以及真实语言语料库中的有效性。因此，我们的模型为研究语言处理和意想不到的普遍智能的物理和生物对应关系提供了一个起点。