In image-grounded text generation, fine-grained representations of the image are considered to be of paramount importance. Most of the current systems incorporate visual features and textual concepts as a sketch of an image. However, plainly inferred representations are usually undesirable in that they are composed of separate components, the relations of which are elusive. In this work, we aim at representing an image with a set of integrated visual regions and corresponding textual concepts. To this end, we build the Mutual Iterative Attention (MIA) module, which integrates correlated visual features and textual concepts, respectively, by aligning the two modalities. We evaluate the proposed approach on the COCO dataset for image captioning. Extensive experiments show that the refined image representations boost the baseline models by up to 12% in terms of CIDEr, demonstrating that our method is effective and generalizes well to a wide range of models.

该论文的研究旨在使用一组集成的视觉区域和相应的文本概念来表示图像，从而反映出特定的语义。为此，研究人员构建了互相迭代注意力（MIA）模块，并将该方法在图像字幕和视觉问答等任务中得到了验证。结果表明，该方法对于图像相关应用具有广泛的泛化能力，并且能将基线模型的性能提升到了一个新的水平。

将视觉区域与文本概念对齐以实现语义驱动图像表征