In this work, we investigate the knowledge learned in the embeddings of
multimodal-BERT models. More specifically, we probe their capabilities of
storing the grammatical structure of linguistic data and the structure learned
over objects in visual data. To reach that goal, we first make the inherent
structure of language and visuals explicit by a dependency parse of the
sentences that describe the image and by the dependencies between the object
regions in the image, respectively. We call this explicit visual structure the
\textit{scene tree}, that is based on the dependency tree of the language
description. Extensive probing experiments show that the multimodal-BERT models
do not encode these scene trees.Code available at
https://github.com/VSJMilewski/multimodal-probes.

本文探讨了多模态 BERT 模型嵌入学习到的知识，并通过依存句法分析语言数据和对象数据之间的依赖关系，构建了显式的可视化场景树，最终实验证明了多模态 BERT 模型不能编码这些场景树。

在多模态 BERT 中发现结构知识

Finding Structural Knowledge in Multimodal-BERT

Search applications often display shortened sentences which must contain
certain query terms and must fit within the space constraints of a user
interface. This work introduces a new transition-based sentence compression
technique developed for such settings. Our query-focused method constructs
length and lexically constrained compressions in linear time, by growing a
subgraph in the dependency parse of a sentence. This theoretically efficient
approach achieves an 11X empirical speedup over baseline ILP methods, while
better reconstructing gold constrained shortenings. Such speedups help
query-focused applications, because users are measurably hindered by interface
lags. Additionally, our technique does not require an ILP solver or a GPU.

本文提出一种基于查询的句子压缩方法，通过在句子的依存语法分析中增长子图，构建长度和词汇受限制的压缩，并且相较于 ILP 方法，取得了 11 倍的实证加速效果，能够帮助查询应用程序提高用户体验。

线性时间内的查询相关句子压缩

Query-focused Sentence Compression in Linear Time

Unlike extractive summarization, abstractive summarization has to fuse
different parts of the source text, which inclines to create fake facts. Our
preliminary study reveals nearly 30% of the outputs from a state-of-the-art
neural summarization system suffer from this problem. While previous
abstractive summarization approaches usually focus on the improvement of
informativeness, we argue that faithfulness is also a vital prerequisite for a
practical abstractive summarization system. To avoid generating fake facts in a
summary, we leverage open information extraction and dependency parse
technologies to extract actual fact descriptions from the source text. The
dual-attention sequence-to-sequence framework is then proposed to force the
generation conditioned on both the source text and the extracted fact
descriptions. Experiments on the Gigaword benchmark dataset demonstrate that
our model can greatly reduce fake summaries by 80%. Notably, the fact
descriptions also bring significant improvement on informativeness since they
often condense the meaning of the source text.

提出了一种基于句法和信息提取的双重注意序列到序列框架，该框架旨在解决抽象汇总中存在的虚假事实问题，并使生成的汇总同时具有信息丰富性和语义准确性。