We propose $\mathbb{VD}$-$\mathbb{GR}$ - a novel visual dialog model that
combines pre-trained language models (LMs) with graph neural networks (GNNs).
Prior works mainly focused on one class of models at the expense of the other,
thus missing out on the opportunity of combining their respective benefits. At
the core of $\mathbb{VD}$-$\mathbb{GR}$ is a novel integration mechanism that
alternates between spatial-temporal multi-modal GNNs and BERT layers, and that
covers three distinct contributions: First, we use multi-modal GNNs to process
the features of each modality (image, question, and dialog history) and exploit
their local structures before performing BERT global attention. Second, we
propose hub-nodes that link to all other nodes within one modality graph,
allowing the model to propagate information from one GNN (modality) to the
other in a cascaded manner. Third, we augment the BERT hidden states with
fine-grained multi-modal GNN features before passing them to the next
$\mathbb{VD}$-$\mathbb{GR}$ layer. Evaluations on VisDial v1.0, VisDial v0.9,
VisDialConv, and VisPro show that $\mathbb{VD}$-$\mathbb{GR}$ achieves new
state-of-the-art results across all four datasets.

我们提出了一种新颖的可视对话模型 V D-G R，它将预训练语言模型与图神经网络相结合，通过空间 - 时间多模式 GNN 和 BERT 层之间的交替集成机制，实现图像、问题和对话历史三个模态的特征处理，并在实验证明 VD-GR 在四个数据集上都达到了最新的最优结果。