We study phrase structure induction from visually-grounded speech. The core
idea is to first segment the speech waveform into sequences of word segments,
and subsequently induce phrase structure using the inferred segment-level
continuous representations. We present the Audio-Visual Neural Syntax Learner
(AV-NSL) that learns phrase structure by listening to audio and looking at
images, without ever being exposed to text. By training on paired images and
spoken captions, AV-NSL exhibits the capability to infer meaningful phrase
structures that are comparable to those derived by naturally-supervised text
parsers, for both English and German. Our findings extend prior work in
unsupervised language acquisition from speech and grounded grammar induction,
and present one approach to bridge the gap between the two topics.

通过从视觉感知语音进行短语结构识别的研究，我们提出了语音 - 视觉神经语法学习器（AV-NSL），通过听音和看图，无需接触文本即可学习短语结构，训练集是配对的图像和口述字幕，AV-NSL 展示出推断有意义的短语结构的能力，对英语和德语都可以与自然监督的文本解析器推导出的结构相媲美，研究结果扩展了无监督语言习得和基于场景的语法识别的以前研究，并提出了一种桥接两个主题的方法。