We study phrase structure induction from visually-grounded speech. The core
idea is to first segment the speech waveform into sequences of word segments,
and subsequently induce phrase structure using the inferred segment-level
continuous representations. We present the Audio-Visual Neural Syntax Learner
(AV-NSL) that learns phrase structure by listening to audio and looking at
images, without ever being exposed to text. By training on paired images and
spoken captions, AV-NSL exhibits the capability to infer meaningful phrase
structures that are comparable to those derived by naturally-supervised text
parsers, for both English and German. Our findings extend prior work in
unsupervised language acquisition from speech and grounded grammar induction,
and present one approach to bridge the gap between the two topics.

通过从视觉感知语音进行短语结构识别的研究，我们提出了语音 - 视觉神经语法学习器（AV-NSL），通过听音和看图，无需接触文本即可学习短语结构，训练集是配对的图像和口述字幕，AV-NSL 展示出推断有意义的短语结构的能力，对英语和德语都可以与自然监督的文本解析器推导出的结构相媲美，研究结果扩展了无监督语言习得和基于场景的语法识别的以前研究，并提出了一种桥接两个主题的方法。

音视觉神经语法学习

Audio-Visual Neural Syntax Acquisition

This work has been motivated by two long term goals: to understand how humans
learn language and to build programs that can understand language. Using a
representation that makes the relevant features explicit is a prerequisite for
successful learning and understanding. Therefore, I chose to represent
relations between individual words explicitly in my model. Lexical attraction
is defined as the likelihood of such relations. I introduce a new class of
probabilistic language models named lexical attraction models which can
represent long distance relations between words and I formalize this new class
of models using information theory.
Within the framework of lexical attraction, I developed an unsupervised
language acquisition program that learns to identify linguistic relations in a
given sentence. The only explicitly represented linguistic knowledge in the
program is lexical attraction. There is no initial grammar or lexicon built in
and the only input is raw text. Learning and processing are interdigitated. The
processor uses the regularities detected by the learner to impose structure on
the input. This structure enables the learner to detect higher level
regularities. Using this bootstrapping procedure, the program was trained on
100 million words of Associated Press material and was able to achieve 60%
precision and 50% recall in finding relations between content-words. Using
knowledge of lexical attraction, the program can identify the correct relations
in syntactically ambiguous sentences such as ``I saw the Statue of Liberty
flying over New York.''

本文介绍了一种新的基于词汇吸引力模型的自动学习自然语言的方式，无需初始化语法和词汇表，只需输入原始文本即可识别词汇间的语言关系，能够准确识别句子中的句法模糊性，并在大规模语料库中达到 60% 的准确率和 50% 的召回率。