Exploiting visual groundings for language understanding has recently been
drawing much attention. In this work, we study visually grounded grammar
induction and learn a constituency parser from both unlabeled text and its
visual groundings. Existing work on this task (Shi et al., 2019) optimizes a
parser via Reinforce and derives the learning signal only from the alignment of
images and sentences. While their model is relatively accurate overall, its
error distribution is very uneven, with low performance on certain constituents
types (e.g., 26.2% recall on verb phrases, VPs) and high on others (e.g., 79.6%
recall on noun phrases, NPs). This is not surprising as the learning signal is
likely insufficient for deriving all aspects of phrase-structure syntax and
gradient estimates are noisy. We show that using an extension of probabilistic
context-free grammar model we can do fully-differentiable end-to-end visually
grounded learning. Additionally, this enables us to complement the image-text
alignment loss with a language modeling objective. On the MSCOCO test captions,
our model establishes a new state of the art, outperforming its non-grounded
version and, thus, confirming the effectiveness of visual groundings in
constituency grammar induction. It also substantially outperforms the previous
grounded model, with largest improvements on more `abstract' categories (e.g.,
+55.1% recall on VPs).

本文将视觉与语言相结合用于语言理解，采用一种基于概率无上下文文法模型的 full-differentiable 端到端视觉理解模型，在 MSCOCO 测试标题上取得了新的最佳效果，证实了视觉基础在短语结构语法归纳中的有效性。

视觉基础复合 PCFG

Visually Grounded Compound PCFGs

We demonstrate that replacing an LSTM encoder with a self-attentive
architecture can lead to improvements to a state-of-the-art discriminative
constituency parser. The use of attention makes explicit the manner in which
information is propagated between different locations in the sentence, which we
use to both analyze our model and propose potential improvements. For example,
we find that separating positional and content information in the encoder can
lead to improved parsing accuracy. Additionally, we evaluate different
approaches for lexical representation. Our parser achieves new state-of-the-art
results for single models trained on the Penn Treebank: 93.55 F1 without the
use of any external data, and 95.13 F1 when using pre-trained word
representations. Our parser also outperforms the previous best-published
accuracy figures on 8 of the 9 languages in the SPMRL dataset.

通过将 LSTM 编码器替换为自注意力机制，能够提高最先进的判别型成分解析器的性能，特别是当用预训练的字词表示时，而且这种方法在 SPMRL 数据集的大部分语言上的性能优于以前的最佳结果。