Humans excel at applying learned behavior to unlearned situations. A crucial
component of this generalization behavior is our ability to compose/decompose a
whole into reusable parts, an attribute known as compositionality. One of the
fundamental questions in robotics concerns this characteristic. "How can
linguistic compositionality be developed concomitantly with sensorimotor skills
through associative learning, particularly when individuals only learn partial
linguistic compositions and their corresponding sensorimotor patterns?" To
address this question, we propose a brain-inspired neural network model that
integrates vision, proprioception, and language into a framework of predictive
coding and active inference, based on the free-energy principle. The
effectiveness and capabilities of this model were assessed through various
simulation experiments conducted with a robot arm. Our results show that
generalization in learning to unlearned verb-noun compositions, is
significantly enhanced when training variations of task composition are
increased. We attribute this to self-organized compositional structures in
linguistic latent state space being influenced significantly by sensorimotor
learning. Ablation studies show that visual attention and working memory are
essential to accurately generate visuo-motor sequences to achieve
linguistically represented goals. These insights advance our understanding of
mechanisms underlying development of compositionality through interactions of
linguistic and sensorimotor experience.

通过基于自由能原理的预测编码和主动推理框架，将视觉、本体感知和语言集成到一个受脑启发的神经网络模型中，通过多种模拟实验及机械臂的测试，研究表明，在增加任务组合的变异性时，对于学习未知的动词 - 名词组合的泛化能力得到了显著提升，这归因于语言潜在状态空间中的自组织组合结构被感觉运动学习显著影响，视觉注意和工作记忆对于准确生成视动序列以实现语言表示目标具有重要作用，这些结果推进了我们对通过语言和感觉运动经验相互作用发展组合性机制的理解。