Having the difficulty of solving the semantic gap between images and texts
for the image captioning task, conventional studies in this area paid some
attention to treating semantic concepts as a bridge between the two modalities
and improved captioning performance accordingly. Although promising results on
concept prediction were obtained, the aforementioned studies normally ignore
the relationship among concepts, which relies on not only objects in the image,
but also word dependencies in the text, so that offers a considerable potential
for improving the process of generating good descriptions. In this paper, we
propose a structured concept predictor (SCP) to predict concepts and their
structures, then we integrate them into captioning, so as to enhance the
contribution of visual signals in this task via concepts and further use their
relations to distinguish cross-modal semantics for better description
generation. Particularly, we design weighted graph convolutional networks
(W-GCN) to depict concept relations driven by word dependencies, and then
learns differentiated contributions from these concepts for following decoding
process. Therefore, our approach captures potential relations among concepts
and discriminatively learns different concepts, so that effectively facilitates
image captioning with inherited information across modalities. Extensive
experiments and their results demonstrate the effectiveness of our approach as
well as each proposed module in this work.

本文介绍了一个基于结构化概念预测器的图像描述生成方法，通过构建带有权重的图卷积网络，可以有效地捕捉概念间关系，并区分性地学习各个概念，从而更好地结合图像和文本信息生成描述。经过广泛实验验证了该方法及其各个模块的有效性。