In this paper, we propose to employ the convolutional neural network (CNN) for learning to answer questions from the image. Our proposed CNN provides an end-to-end framework for learning not only the image representation, the composition model for question, but also the inter-modal int