This paper presents an approach for answering fill-in-the-blank multiple
choice questions from the Visual Madlibs dataset. Instead of generic and
commonly used representations trained on the ImageNet classification task, our
approach employs a combination of networks trained for specialized tasks such
as scene recognition, person activity classification, and attribute prediction.
We also present a method for localizing phrases from candidate answers in order
to provide spatial support for feature extraction. We map each of these
features, together with candidate answers, to a joint embedding space through
normalized canonical correlation analysis (nCCA). Finally, we solve an
optimization problem to learn to combine scores from nCCA models trained on
multiple cues to select the best answer. Extensive experimental results show a
significant improvement over the previous state of the art and confirm that
answering questions from a wide range of types benefits from examining a
variety of image cues and carefully choosing the spatial support for feature
extraction.

本论文提出一种从视觉 Madlibs 数据集回答填空多项选择题的方法，通过使用针对特定任务训练的网络的组合，结合特征提取的空间支持，将每个特征与候选答案映射到联合嵌入空间并解决优化问题，以学习组合来自多个提示的 CCA 模型的分数以选择最佳答案。实验结果表明，与现有技术相比，本方法取得了显著的改进，且在处理各种问题类型时，使用各种图像提示并仔细选择特征提取的空间支持有益。