Image captioning is shown to be able to achieve a better performance by using
scene graphs to represent the relations of objects in the image. The current
captioning encoders generally use a Graph Convolutional Net (GCN) to represent
the relation information and merge it with the object region features via
concatenation or convolution to get the final input for sentence decoding.
However, the GCN-based encoders in the existing methods are less effective for
captioning due to two reasons. First, using the image captioning as the
objective (i.e., Maximum Likelihood Estimation) rather than a relation-centric
loss cannot fully explore the potential of the encoder. Second, using a
pre-trained model instead of the encoder itself to extract the relationships is
not flexible and cannot contribute to the explainability of the model. To
improve the quality of image captioning, we propose a novel architecture
ReFormer -- a RElational transFORMER to generate features with relation
information embedded and to explicitly express the pair-wise relationships
between objects in the image. ReFormer incorporates the objective of scene
graph generation with that of image captioning using one modified Transformer
model. This design allows ReFormer to generate not only better image captions
with the bene-fit of extracting strong relational image features, but also
scene graphs to explicitly describe the pair-wise relation-ships. Experiments
on publicly available datasets show that our model significantly outperforms
state-of-the-art methods on image captioning and scene graph generation

提出了一种新的神经网络结构 ReFormer，该网络基于 Transformer 模型，融合了场景图生成目标与图像描述目标，实现了对图像对象间关系的显式建模，并能够同时生成高质量的图像描述和场景图，实验结果证明 ReFormer 在图像描述与场景图生成方面均优于当前的最新模型。