Unsupervised image captioning is a challenging task that aims at generating
captions without the supervision of image-sentence pairs, but only with images
and sentences drawn from different sources and object labels detected from the
images. In previous work, pseudo-captions, i.e., sentences that contain the
detected object labels, were assigned to a given image. The focus of the
previous work was on the alignment of input images and pseudo-captions at the
sentence level. However, pseudo-captions contain many words that are irrelevant
to a given image. In this work, we investigate the effect of removing
mismatched words from image-sentence alignment to determine how they make this
task difficult. We propose a simple gating mechanism that is trained to align
image features with only the most reliable words in pseudo-captions: the
detected object labels. The experimental results show that our proposed method
outperforms the previous methods without introducing complex sentence-level
learning objectives. Combined with the sentence-level alignment method of
previous work, our method further improves its performance. These results
confirm the importance of careful alignment in word-level details.

本文主要探讨了通过提出一个简单的门控机制来从伪字幕中选择可靠的单词以提高图像字幕生成任务的准确性，并证明了在单词级别详细对齐方面的慎重对齐的重要性。

无监督图像字幕生成中，消除图像与伪字幕之间的单词级误对齐

Removing Word-Level Spurious Alignment between Images and  Pseudo-Captions in Unsupervised Image Captioning

Unsupervised image captioning with no annotations is an emerging challenge in
computer vision, where the existing arts usually adopt GAN (Generative
Adversarial Networks) models. In this paper, we propose a novel memory-based
network rather than GAN, named Recurrent Relational Memory Network ($R^2M$).
Unlike complicated and sensitive adversarial learning that non-ideally performs
for long sentence generation, $R^2M$ implements a concepts-to-sentence memory
translator through two-stage memory mechanisms: fusion and recurrent memories,
correlating the relational reasoning between common visual concepts and the
generated words for long periods. $R^2M$ encodes visual context through
unsupervised training on images, while enabling the memory to learn from
irrelevant textual corpus via supervised fashion. Our solution enjoys less
learnable parameters and higher computational efficiency than GAN-based
methods, which heavily bear parameter sensitivity. We experimentally validate
the superiority of $R^2M$ than state-of-the-arts on all benchmark datasets.

本文提出了一种基于记忆机制的无监督图像描述模型 —— 循环关系记忆网络 (R2M)，相较于 GAN 模型，该方法克服了句子生成中复杂和敏感的对抗学习，并且通过融合和循环两种记忆机制实现了从通用可视化概念到自然语言的翻译；我们在多项基准数据集上的实验证明，R2M 在参数少、计算效率高等各方面均优于目前最先进的技术。