The primary focus of recent work with largescale transformers has been on
optimizing the amount of information packed into the model's parameters. In
this work, we ask a different question: Can multimodal transformers leverage
explicit knowledge in their reasoning? Existing, primarily unimodal, methods
have explored approaches under the paradigm of knowledge retrieval followed by
answer prediction, but leave open questions about the quality and relevance of
the retrieved knowledge used, and how the reasoning processes over implicit and
explicit knowledge should be integrated. To address these challenges, we
propose a novel model - Knowledge Augmented Transformer (KAT) - which achieves
a strong state-of-the-art result (+6 points absolute) on the open-domain
multimodal task of OK-VQA. Our approach integrates implicit and explicit
knowledge in an end to end encoder-decoder architecture, while still jointly
reasoning over both knowledge sources during answer generation. An additional
benefit of explicit knowledge integration is seen in improved interpretability
of model predictions in our analysis.

本文介绍了一种新型多模态转换方法 - 知识增强变压器（KAT），该方法在 OK-VQA 上取得了强有力的最新结果，同时还提高了模型预测的可解释性。

KAT：一种知识增强的视觉与语言 Transformer 模型

KAT: A Knowledge Augmented Transformer for Vision-and-Language

Visual Question Answering (VQA) in its ideal form lets us study reasoning in
the joint space of vision and language and serves as a proxy for the AI task of
scene understanding. However, most VQA benchmarks to date are focused on
questions such as simple counting, visual attributes, and object detection that
do not require reasoning or knowledge beyond what is in the image. In this
paper, we address the task of knowledge-based visual question answering and
provide a benchmark, called OK-VQA, where the image content is not sufficient
to answer the questions, encouraging methods that rely on external knowledge
resources. Our new dataset includes more than 14,000 questions that require
external knowledge to answer. We show that the performance of the
state-of-the-art VQA models degrades drastically in this new setting. Our
analysis shows that our knowledge-based VQA task is diverse, difficult, and
large compared to previous knowledge-based VQA datasets. We hope that this
dataset enables researchers to open up new avenues for research in this domain.
See this http URL to download and browse the dataset.

本文提出了名为 OK-VQA 的基于知识的视觉问答基准，要求使用外部知识资源来回答问题，该任务是多样的、困难的和大型的，与以前的基于知识的 VQA 数据集相比具有更高的难度。