In this paper, we present the mQA model, which is able to answer questions
about the content of an image. The answer can be a sentence, a phrase or a
single word. Our model contains four components: a Long Short-Term Memory
(LSTM) to extract the question representation, a Convolutional Neural Network
(CNN) to extract the visual representation, an LSTM for storing the linguistic
context in an answer, and a fusing component to combine the information from
the first three components and generate the answer. We construct a Freestyle
Multilingual Image Question Answering (FM-IQA) dataset to train and evaluate
our mQA model. It contains over 150,000 images and 310,000 freestyle Chinese
question-answer pairs and their English translations. The quality of the
generated answers of our mQA model on this dataset is evaluated by human judges
through a Turing Test. Specifically, we mix the answers provided by humans and
our model. The human judges need to distinguish our model from the human. They
will also provide a score (i.e. 0, 1, 2, the larger the better) indicating the
quality of the answer. We propose strategies to monitor the quality of this
evaluation process. The experiments show that in 64.7% of cases, the human
judges cannot distinguish our model from humans. The average score is 1.454
(1.918 for human). The details of this work, including the FM-IQA dataset, can
be found on the project page: this http URL

本文介绍了基于 LSTM 和卷积神经网络的 mQA 模型，其可针对图像内容回答包括中英文在内的多语言问题，并使用 Freestyle Multilingual Image Question Answering (FM-IQA) 数据集进行评估，结果表明该模型在 64.7% 的情况下与人类回答无异，评分为 1.454。