The goal of unpaired image captioning (UIC) is to describe images without
using image-caption pairs in the training phase. Although challenging, we
except the task can be accomplished by leveraging a training set of images
aligned with visual concepts. Most existing studies use off-the-shelf
algorithms to obtain the visual concepts because the Bounding Box (BBox) labels
or relationship-triplet labels used for the training are expensive to acquire.
In order to resolve the problem in expensive annotations, we propose a novel
approach to achieve cost-effective UIC. Specifically, we adopt image-level
labels for the optimization of the UIC model in a weakly-supervised manner. For
each image, we assume that only the image-level labels are available without
specific locations and numbers. The image-level labels are utilized to train a
weakly-supervised object recognition model to extract object information (e.g.,
instance) in an image, and the extracted instances are adopted to infer the
relationships among different objects based on an enhanced graph neural network
(GNN). The proposed approach achieves comparable or even better performance
compared with previous methods without the expensive cost of annotations.
Furthermore, we design an unrecognized object (UnO) loss combined with a visual
concept reward to improve the alignment of the inferred object and relationship
information with the images. It can effectively alleviate the issue encountered
by existing UIC models about generating sentences with nonexistent objects. To
the best of our knowledge, this is the first attempt to solve the problem of
Weakly-Supervised visual concept recognition for UIC (WS-UIC) based only on
image-level labels. Extensive experiments have been carried out to demonstrate
that the proposed WS-UIC model achieves inspiring results on the COCO dataset
while significantly reducing the cost of labeling.

通过利用图像级别标签进行弱监督学习，提出解决成本昂贵的标注问题的无配对图像字幕生成模型，在视觉概念识别和图神经网络方面都取得了较好的表现。

基于图像级弱监督视觉概念识别的非配对图像字幕生成

Unpaired Image Captioning by Image-level Weakly-Supervised Visual Concept Recognition

Recently, image captioning has aroused great interest in both academic and
industrial worlds. Most existing systems are built upon large-scale datasets
consisting of image-sentence pairs, which, however, are time-consuming to
construct. In addition, even for the most advanced image captioning systems, it
is still difficult to realize deep image understanding. In this work, we
achieve unpaired image captioning by bridging the vision and the language
domains with high-level semantic information. The motivation stems from the
fact that the semantic concepts with the same modality can be extracted from
both images and descriptions. To further improve the quality of captions
generated by the model, we propose the Semantic Relationship Explorer, which
explores the relationships between semantic concepts for better understanding
of the image. Extensive experiments on MSCOCO dataset show that we can generate
desirable captions without paired datasets. Furthermore, the proposed approach
boosts five strong baselines under the paired setting, where the most
significant improvement in CIDEr score reaches 8%, demonstrating that it is
effective and generalizes well to a wide range of models.

本文提出了一种跨越视觉和语言领域的高级语义信息桥接技术，以实现无配对图像说明，并引入了基于语义概念关系探索的解决方案 ——Semantic Relationship Explorer，能够有效地提高图像说明的质量，并且在 MSCOCO 数据集下总体得分比配对数据集高出 8%。