Visual dialog is a vision-language task where an agent needs to answer a
series of questions grounded in an image based on the understanding of the
dialog history and the image. The occurrences of coreference relations in the
dialog makes it a more challenging task than visual question-answering. Most
previous works have focused on learning better multi-modal representations or
on exploring different ways of fusing visual and language features, while the
coreferences in the dialog are mainly ignored. In this paper, based on
linguistic knowledge and discourse features of human dialog we propose two soft
constraints that can improve the model's ability of resolving coreferences in
dialog in an unsupervised way. Experimental results on the VisDial v1.0 dataset
shows that our model, which integrates two novel and linguistically inspired
soft constraints in a deep transformer neural architecture, obtains new
state-of-the-art performance in terms of recall at 1 and other evaluation
metrics compared to current existing models and this without pretraining on
other vision-language datasets. Our qualitative results also demonstrate the
effectiveness of the method that we propose.

本文提出两个基于人类对话语言知识和语境特征的软约束来改进模型的对话代词消解能力，将这两个约束嵌入到深度 transformer 神经架构中。实验结果显示出了本模型在 VisDial v1.0 数据集上相比于现有模型的性能提升，无需预训练其他视觉语言数据集。

模拟视觉对话中的指代关系

Modeling Coreference Relations in Visual Dialog

In this work, we propose to use linguistic annotations as a basis for a
\textit{Discourse-Aware Semantic Self-Attention} encoder that we employ for
reading comprehension on long narrative texts. We extract relations between
discourse units, events and their arguments as well as coreferring mentions,
using available annotation tools. Our empirical evaluation shows that the
investigated structures improve the overall performance, especially
intra-sentential and cross-sentential discourse relations, sentence-internal
semantic role relations, and long-distance coreference relations. We show that
dedicating self-attention heads to intra-sentential relations and relations
connecting neighboring sentences is beneficial for finding answers to questions
in longer contexts. Our findings encourage the use of discourse-semantic
annotations to enhance the generalization capacity of self-attention models for
reading comprehension.

使用语言学注释作为基础，提出结构改进并应用于长篇叙述文本的阅读理解中，通过提取段落单元之间的关系、事件与其参数以及代词提及之间的关系来提高模型的性能，尤其是在语境内部的语义角色关系、句内关系、长距离代词关系中。实验结果表明，在需考虑到句内、句间关系的长文本中，将注意力集中在这些关系上可以提高模型的性能。研究结果表明，使用语篇 - 语义注释可以增强自注意力模型在阅读理解中的泛化能力。

针对叙事阅读理解的语境感知语义自注意力

Discourse-Aware Semantic Self-Attention for Narrative Reading  Comprehension

Training recurrent neural networks to model long term dependencies is
difficult. Hence, we propose to use external linguistic knowledge as an
explicit signal to inform the model which memories it should utilize.
Specifically, external knowledge is used to augment a sequence with typed edges
between arbitrarily distant elements, and the resulting graph is decomposed
into directed acyclic subgraphs. We introduce a model that encodes such graphs
as explicit memory in recurrent neural networks, and use it to model
coreference relations in text. We apply our model to several text comprehension
tasks and achieve new state-of-the-art results on all considered benchmarks,
including CNN, bAbi, and LAMBADA. On the bAbi QA tasks, our model solves 15 out
of the 20 tasks with only 1000 training examples per task. Analysis of the
learned representations further demonstrates the ability of our model to encode
fine-grained entity information across a document.

通过使用外部语言知识来作为 RNN 模型的显式信号，将文本序列图形化表示并进行无环子图分解，以实现 RNN 模型对长期依赖关系的建模，并应用于文本理解任务，实现新的 SOTA 效果。