Outside-knowledge visual question answering is a challenging task that
requires both the acquisition and the use of open-ended real-world knowledge.
Some existing solutions draw external knowledge into the cross-modality space
which overlooks the much vaster textual knowledge in natural-language space,
while others transform the image into a text that further fuses with the
textual knowledge into the natural-language space and completely abandons the
use of visual features. In this paper, we are inspired to constrain the
cross-modality space into the same space of natural-language space which makes
the visual features preserved directly, and the model still benefits from the
vast knowledge in natural-language space. To this end, we propose a novel
framework consisting of a multimodal encoder, a textual encoder and an answer
decoder. Such structure allows us to introduce more types of knowledge
including explicit and implicit multimodal and textual knowledge. Extensive
experiments validate the superiority of the proposed method which outperforms
the state-of-the-art by 6.17% accuracy. We also conduct comprehensive ablations
of each component, and systematically study the roles of varying types of
knowledge. Codes and knowledge data can be found at
this https URL

本文提出一种包含多模态编码器、文本编码器和答案解码器的新型框架，将跨模态空间约束到自然语言空间中，使得视觉特征直接保留在其中，并且从自然语言空间中获得更多的类型知识，实验证明其在多数情况下表现优异。