We study visual question answering in a setting where the answer has to be
mined from a pool of relevant and irrelevant images given as a context. For
such a setting, a model must first retrieve relevant images from the pool and
answer the question from these retrieved images. We refer to this problem as
retrieval-based visual question answering (or RETVQA in short). The RETVQA is
distinctively different and more challenging than the traditionally-studied
Visual Question Answering (VQA), where a given question has to be answered with
a single relevant image in context. Towards solving the RETVQA task, we propose
a unified Multi Image BART (MI-BART) that takes a question and retrieved images
using our relevance encoder for free-form fluent answer generation. Further, we
introduce the largest dataset in this space, namely RETVQA, which has the
following salient features: multi-image and retrieval requirement for VQA,
metadata-independent questions over a pool of heterogeneous images, expecting a
mix of classification-oriented and open-ended generative answers. Our proposed
framework achieves an accuracy of 76.5% and a fluency of 79.3% on the proposed
dataset, namely RETVQA and also outperforms state-of-the-art methods by 4.9%
and 11.8% on the image segment of the publicly available WebQA dataset on the
accuracy and fluency metrics, respectively.

本文介绍了一种针对多张图片中远程问答的视觉问答模型，并提出了一个名为 RETVQA 的具有多图和检索需求的新数据集，使用 MI-BART 模型在该数据集上取得了良好的表现。

从图像池中挖掘答案：走向基于检索的视觉问答

Answer Mining from a Pool of Images: Towards Retrieval-Based Visual  Question Answering

Intelligent personal assistant systems that are able to have multi-turn
conversations with human users are becoming increasingly popular. Most previous
research has been focused on using either retrieval-based or generation-based
methods to develop such systems. Retrieval-based methods have the advantage of
returning fluent and informative responses with great diversity. However, the
performance of the methods is limited by the size of the response repository.
On the other hand, generation-based methods can produce highly coherent
responses on any topics. But the generated responses are often generic and not
informative due to the lack of grounding knowledge. In this paper, we propose a
hybrid neural conversation model that combines the merits of both response
retrieval and generation methods. Experimental results on Twitter and
Foursquare data show that the proposed model outperforms both retrieval-based
methods and generation-based methods (including a recently proposed
knowledge-grounded neural conversation model) under both automatic evaluation
metrics and human evaluation. We hope that the findings in this study provide
new insights on how to integrate text retrieval and text generation models for
building conversation systems.

本文提出了一种混合神经对话模型，其结合了检索和生成方法的优点，并经实验证明其在 Twitter 和 Foursquare 数据上优于检索方法和生成方法（包括最近提出的基于知识的神经对话模型），这将为构建对话系统提供新的思路。