We address a question answering task on real-world images that is set up as a
Visual Turing Test. By combining latest advances in image representation and
natural language processing, we propose Neural-Image-QA, an end-to-end
formulation to this problem for which all parts are trained jointly. In
contrast to previous efforts, we are facing a multi-modal problem where the
language output (answer) is conditioned on visual and natural language input
(image and question). Our approach Neural-Image-QA doubles the performance of
the previous best approach on this problem. We provide additional insights into
the problem by analyzing how much information is contained only in the language
part for which we provide a new human baseline. To study human consensus, which
is related to the ambiguities inherent in this challenging task, we propose two
novel metrics and collect additional answers which extends the original DAQUAR
dataset to DAQUAR-Consensus.

利用图像表示和自然语言处理的最新进展，我们提出了一种端到端形式的 Neural-Image-QA 解决方案来回答真实世界图像上的问题，我们面临一个多模态问题，其中语言输出（答案）是基于视觉和自然语言输入（图像和问题）的，我们的方法可以使已有研究成果的表现效果成倍增加，并提供了关于该问题的额外见解，包括研究人类共识的两个新指标并扩展了初始数据集到 DAQUAR-Consensus。

问你神经元：基于神经网络的图像问答系统

Ask Your Neurons: A Neural-based Approach to Answering Questions about  Images

Automatically describing an image with a sentence is a long-standing
challenge in computer vision and natural language processing. Due to recent
progress in object detection, attribute classification, action recognition,
etc., there is renewed interest in this area. However, evaluating the quality
of descriptions has proven to be challenging. We propose a novel paradigm for
evaluating image descriptions that uses human consensus. This paradigm consists
of three main parts: a new triplet-based method of collecting human annotations
to measure consensus, a new automated metric (CIDEr) that captures consensus,
and two new datasets: PASCAL-50S and ABSTRACT-50S that contain 50 sentences
describing each image. Our simple metric captures human judgment of consensus
better than existing metrics across sentences generated by various sources. We
also evaluate five state-of-the-art image description approaches using this new
protocol and provide a benchmark for future comparisons. A version of CIDEr
named CIDEr-D is available as a part of MS COCO evaluation server to enable
systematic evaluation and benchmarking.

本文提出了一种基于人类共识的评估图像描述的新方法，包括新的基于三元组的人类注释方法、一种捕捉共识的新自动化指标（CIDEr）和包含 50 个对每个图像进行描述的句子的两个新数据集（PASCAL-50S 和 ABSTRACT-50S）。使用这种新协议评估了五种最先进的图像描述方法，并提供了未来比较的基准。