Generating image descriptions in different languages is essential to satisfy
users worldwide. However, it is prohibitively expensive to collect large-scale
paired image-caption dataset for every target language which is critical for
training descent image captioning models. Previous works tackle the unpaired
cross-lingual image captioning problem through a pivot language, which is with
the help of paired image-caption data in the pivot language and pivot-to-target
machine translation models. However, such language-pivoted approach suffers
from inaccuracy brought by the pivot-to-target translation, including
disfluency and visual irrelevancy errors. In this paper, we propose to generate
cross-lingual image captions with self-supervised rewards in the reinforcement
learning framework to alleviate these two types of errors. We employ
self-supervision from mono-lingual corpus in the target language to provide
fluency reward, and propose a multi-level visual semantic matching model to
provide both sentence-level and concept-level visual relevancy rewards. We
conduct extensive experiments for unpaired cross-lingual image captioning in
both English and Chinese respectively on two widely used image caption corpora.
The proposed approach achieves significant performance improvement over
state-of-the-art methods.

本文介绍一种利用增强学习的自监督奖励和多级视觉语义匹配模型来生成跨语言图像描述的方法，在英语和汉语两种语言测试中分别取得了显著的性能提升。