Change captioning is to use a natural language sentence to describe the
fine-grained disagreement between two similar images. Viewpoint change is the
most typical distractor in this task, because it changes the scale and location
of the objects and overwhelms the representation of real change. In this paper,
we propose a Relation-embedded Representation Reconstruction Network (R$^3$Net)
to explicitly distinguish the real change from the large amount of clutter and
irrelevant changes. Specifically, a relation-embedded module is first devised
to explore potential changed objects in the large amount of clutter. Then,
based on the semantic similarities of corresponding locations in the two
images, a representation reconstruction module (RRM) is designed to learn the
reconstruction representation and further model the difference representation.
Besides, we introduce a syntactic skeleton predictor (SSP) to enhance the
semantic interaction between change localization and caption generation.
Extensive experiments show that the proposed method achieves the
state-of-the-art results on two public datasets.

本文提出了一个基于关系嵌入的表示重建网络（R3Net）来明确区分真实变化和大量混乱和无关变化，并引入了外部语法骨架预测器（SSP）来增强变化定位和字幕生成之间的语义交互，实验结果表明，所提出的方法在两个公共数据集上达到了最先进的水平。

R$^3$Net: 基于关系嵌入的表示重建网络用于变化描述

R$^3$Net:Relation-embedded Representation Reconstruction Network for  Change Captioning

Change captioning tasks aim to detect changes in image pairs observed before
and after a scene change and generate a natural language description of the
changes. Existing change captioning studies have mainly focused on a single
change.However, detecting and describing multiple changed parts in image pairs
is essential for enhancing adaptability to complex scenarios. We solve the
above issues from three aspects: (i) We propose a simulation-based multi-change
captioning dataset; (ii) We benchmark existing state-of-the-art methods of
single change captioning on multi-change captioning; (iii) We further propose
Multi-Change Captioning transformers (MCCFormers) that identify change regions
by densely correlating different regions in image pairs and dynamically
determines the related change regions with words in sentences. The proposed
method obtained the highest scores on four conventional change captioning
evaluation metrics for multi-change captioning. Additionally, our proposed
method can separate attention maps for each change and performs well with
respect to change localization. Moreover, the proposed framework outperformed
the previous state-of-the-art methods on an existing change captioning
benchmark, CLEVR-Change, by a large margin (+6.1 on BLEU-4 and +9.7 on CIDEr
scores), indicating its general ability in change captioning tasks.

本研究提出基于模拟的多变化字幕数据集并将现有单个变化字幕学习方法应用于多变化字幕任务上，此外，我们还提出 MCCFormers 方法以识别图像对中的变化区域并动态确定与句子中的单词相关的变化区域，该方法在多变化字幕评估指标方面表现出最高得分，在变化局部化方面也表现出色，且优于现有的变化字幕基准。