The problem of Visual Question Answering (VQA) requires joint image and language understanding to answer a question about a given photograph. Recent approaches have applied deep image captioning methods based on recurrent LSTM networks to this problem, but have failed to model spatial inference. In this paper, we propose a memory network with spatial attention for the VQA task. Memory networks are recurrent neural networks with an explicit attention mechanism that selects certain parts of the information stored in memory. We store neuron activations from different spatial receptive fields in the memory, and use the question to choose relevant regions for computing the answer. We experiment with spatial attention architectures that use different question representations to choose regions, and also show that two attention steps (hops) obtain improved results compared to a single step. To understand the inference process learned by the network, we design synthetic questions that specifically require spatial inference and visualize the attention weights. We evaluate our model on two published visual question answering datasets, DAQUAR and VQA, and obtain promising results.

该论文探讨了图像和语言理解的问题，提出了一种基于神经网络的空间记忆网络模型，通过注意力机制进行推理，并在两个视觉问答数据集上取得了改进的结果。

问、看、答：探索基于问题引导的空间关注用于视觉问答