Humans encode information into sounds by controlling articulators and decode
information from sounds using the auditory apparatus. This paper introduces
CiwaGAN, a model of human spoken language acquisition that combines
unsupervised articulatory modeling with an unsupervised model of information
exchange through the auditory modality. While prior research includes
unsupervised articulatory modeling and information exchange separately, our
model is the first to combine the two components. The paper also proposes an
improved articulatory model with more interpretable internal representations.
The proposed CiwaGAN model is the most realistic approximation of human spoken
language acquisition using deep learning. As such, it is useful for cognitively
plausible simulations of the human speech act.

这篇论文介绍了 CiwaGAN 模型，它是结合了无监督的发音建模和通过听觉模态进行信息交流的人类口语习得模型。该模型是第一个将这两个组件结合到一起的研究，同时还提出了一个具有更易解释的内部表示的改进发音模型。提出的 CiwaGAN 模型是使用深度学习对人类口语习得进行最逼真的近似模型，对认知合理的人类言语行为模拟非常有用。

CiwaGAN：音韵信息交流

CiwaGAN: Articulatory information exchange

Recent computational models of the acquisition of spoken language via
grounding in perception exploit associations between the spoken and visual
modalities and learn to represent speech and visual data in a joint vector
space. A major unresolved issue from the point of ecological validity is the
training data, typically consisting of images or videos paired with spoken
descriptions of what is depicted. Such a setup guarantees an unrealistically
strong correlation between speech and the visual data. In the real world the
coupling between the linguistic and the visual modality is loose, and often
confounded by correlations with non-semantic aspects of the speech signal. Here
we address this shortcoming by using a dataset based on the children's cartoon
Peppa Pig. We train a simple bi-modal architecture on the portion of the data
consisting of dialog between characters, and evaluate on segments containing
descriptive narrations. Despite the weak and confounded signal in this training
data our model succeeds at learning aspects of the visual semantics of spoken
language.

通过对儿童动画片数据集的使用，研究利用视觉和语音数据在联合矢量空间中表示话语与视觉信息的双向架构，以解决该领域的研究中存在的数据训练不足的问题。