Unpaired cross-lingual image captioning has long suffered from irrelevancy
and disfluency issues, due to the inconsistencies of the semantic scene and
syntax attributes during transfer. In this work, we propose to address the
above problems by incorporating the scene graph (SG) structures and the
syntactic constituency (SC) trees. Our captioner contains the semantic
structure-guided image-to-pivot captioning and the syntactic structure-guided
pivot-to-target translation, two of which are joined via pivot language. We
then take the SG and SC structures as pivoting, performing cross-modal semantic
structure alignment and cross-lingual syntactic structure alignment learning.
We further introduce cross-lingual&cross-modal back-translation training to
fully align the captioning and translation stages. Experiments on
English-Chinese transfers show that our model shows great superiority in
improving captioning relevancy and fluency.

本文提出引入场景图和句法树来解决跨语言图像描述中存在的不相关性和表达不流畅的问题，并介绍了一种跨语言和跨模态的后向翻译训练方法，实现图像到最终描述的完全对齐，实验结果表明该模型在提高图像描述的相关性和流畅性方面表现出很大的优势。