Explicit Caption Editing (ECE) -- refining reference image captions through a
sequence of explicit edit operations (e.g., KEEP, DETELE) -- has raised
significant attention due to its explainable and human-like nature. After
training with carefully designed reference and ground-truth caption pairs,
state-of-the-art ECE models exhibit limited generalization ability beyond the
original training data distribution, i.e., they are tailored to refine content
details only in in-domain samples but fail to correct errors in out-of-domain
samples. To this end, we propose a new Diffusion-based Explicit Caption editing
method: DECap. Specifically, we reformulate the ECE task as a denoising process
under the diffusion mechanism, and introduce innovative edit-based noising and
denoising processes. Thanks to this design, the noising process can help to
eliminate the need for meticulous paired data selection by directly introducing
word-level noises for training, learning diverse distribution over input
reference caption. The denoising process involves the explicit predictions of
edit operations and corresponding content words, refining reference captions
through iterative step-wise editing. To further efficiently implement our
diffusion process and improve the inference speed, DECap discards the prevalent
multi-stage design and directly generates edit operations and content words
simultaneously. Extensive ablations have demonstrated the strong generalization
ability of DECap in various scenarios. More interestingly, it even shows great
potential in improving the quality and controllability of caption generation.

Diffusion-based Explicit Caption editing (DECap) is proposed as a method to refine reference captions by formulating the task as a denoising process, enabling strong generalization ability and improving the quality and controllability of caption generation.

DECap: 通过扩散机制实现广义显式标题编辑

DECap: Towards Generalized Explicit Caption Editing via Diffusion  Mechanism

Given an image and a reference caption, the image caption editing task aims
to correct the misalignment errors and generate a refined caption. However, all
existing caption editing works are implicit models, ie, they directly produce
the refined captions without explicit connections to the reference captions. In
this paper, we introduce a new task: Explicit Caption Editing (ECE). ECE models
explicitly generate a sequence of edit operations, and this edit operation
sequence can translate the reference caption into a refined one. Compared to
the implicit editing, ECE has multiple advantages: 1) Explainable: it can trace
the whole editing path. 2) Editing Efficient: it only needs to modify a few
words. 3) Human-like: it resembles the way that humans perform caption editing,
and tries to keep original sentence structures. To solve this new task, we
propose the first ECE model: TIger. TIger is a non-autoregressive
transformer-based model, consisting of three modules: Tagger_del, Tagger_add,
and Inserter. Specifically, Tagger_del decides whether each word should be
preserved or not, Tagger_add decides where to add new words, and Inserter
predicts the specific word for adding. To further facilitate ECE research, we
propose two new ECE benchmarks by re-organizing two existing datasets, dubbed
COCO-EE and Flickr30K-EE, respectively. Extensive ablations on both two
benchmarks have demonstrated the effectiveness of TIger.

本文介绍了一个新的任务：显式标题编辑，并提出了一种基于非自回归变压器的 ECE 模型 TIger。在两个 ECE 基准测试上的广泛实验表明 TIger 的有效性。