Multi-change captioning aims to describe complex and coupled changes within
an image pair in natural language. Compared with single-change captioning, this
task requires the model to have higher-level cognition ability to reason an
arbitrary number of changes. In this paper, we propose a novel context-aware
difference distilling (CARD) network to capture all genuine changes for
yielding sentences. Given an image pair, CARD first decouples context features
that aggregate all similar/dissimilar semantics, termed common/difference
context features. Then, the consistency and independence constraints are
designed to guarantee the alignment/discrepancy of common/difference context
features. Further, the common context features guide the model to mine locally
unchanged features, which are subtracted from the pair to distill locally
difference features. Next, the difference context features augment the locally
difference features to ensure that all changes are distilled. In this way, we
obtain an omni-representation of all changes, which is translated into
linguistic sentences by a transformer decoder. Extensive experiments on three
public datasets show CARD performs favourably against state-of-the-art
methods.The code is available at this https URL

该研究提出了一种新颖的上下文感知差异提取（CARD）网络，用于描述自然语言中图像对中的复杂和耦合变化，通过捕捉所有真实的变化来生成句子，并在三个公共数据集上展示了优于现有方法的性能。

面向上下文的多变动字幕中的差异提取

Context-aware Difference Distilling for Multi-change Captioning

Change captioning tasks aim to detect changes in image pairs observed before
and after a scene change and generate a natural language description of the
changes. Existing change captioning studies have mainly focused on a single
change.However, detecting and describing multiple changed parts in image pairs
is essential for enhancing adaptability to complex scenarios. We solve the
above issues from three aspects: (i) We propose a simulation-based multi-change
captioning dataset; (ii) We benchmark existing state-of-the-art methods of
single change captioning on multi-change captioning; (iii) We further propose
Multi-Change Captioning transformers (MCCFormers) that identify change regions
by densely correlating different regions in image pairs and dynamically
determines the related change regions with words in sentences. The proposed
method obtained the highest scores on four conventional change captioning
evaluation metrics for multi-change captioning. Additionally, our proposed
method can separate attention maps for each change and performs well with
respect to change localization. Moreover, the proposed framework outperformed
the previous state-of-the-art methods on an existing change captioning
benchmark, CLEVR-Change, by a large margin (+6.1 on BLEU-4 and +9.7 on CIDEr
scores), indicating its general ability in change captioning tasks.

本研究提出基于模拟的多变化字幕数据集并将现有单个变化字幕学习方法应用于多变化字幕任务上，此外，我们还提出 MCCFormers 方法以识别图像对中的变化区域并动态确定与句子中的单词相关的变化区域，该方法在多变化字幕评估指标方面表现出最高得分，在变化局部化方面也表现出色，且优于现有的变化字幕基准。