Image captioning attempts to generate a sentence composed of several
linguistic words, which are used to describe objects, attributes, and
interactions in an image, denoted as visual semantic units in this paper. Based
on this view, we propose to explicitly model the object interactions in
semantics and geometry based on Graph Convolutional Networks (GCNs), and fully
exploit the alignment between linguistic words and visual semantic units for
image captioning. Particularly, we construct a semantic graph and a geometry
graph, where each node corresponds to a visual semantic unit, i.e., an object,
an attribute, or a semantic (geometrical) interaction between two objects.
Accordingly, the semantic (geometrical) context-aware embeddings for each unit
are obtained through the corresponding GCN learning processers. At each time
step, a context gated attention module takes as inputs the embeddings of the
visual semantic units and hierarchically align the current word with these
units by first deciding which type of visual semantic unit (object, attribute,
or interaction) the current word is about, and then finding the most correlated
visual semantic units under this type. Extensive experiments are conducted on
the challenging MS-COCO image captioning dataset, and superior results are
reported when comparing to state-of-the-art approaches.

本文提出了一种结合图卷积网络（GCN）的视觉语义单元对象相互作用的图形语义和几何建模方法，利用上下文门控注意力模块将当前单词与视觉语义单元对齐，针对 MS-COCO 图像字幕数据集，与现有方法相比报告了更好的结果。