AI-synthesized text and images have gained significant attention,
particularly due to the widespread dissemination of multi-modal manipulations
on the internet, which has resulted in numerous negative impacts on society.
Existing methods for multi-modal manipulation detection and grounding primarily
focus on fusing vision-language features to make predictions, while overlooking
the importance of modality-specific features, leading to sub-optimal results.
In this paper, we construct a simple and novel transformer-based framework for
multi-modal manipulation detection and grounding tasks. Our framework
simultaneously explores modality-specific features while preserving the
capability for multi-modal alignment. To achieve this, we introduce
visual/language pre-trained encoders and dual-branch cross-attention (DCA) to
extract and fuse modality-unique features. Furthermore, we design decoupled
fine-grained classifiers (DFC) to enhance modality-specific feature mining and
mitigate modality competition. Moreover, we propose an implicit manipulation
query (IMQ) that adaptively aggregates global contextual cues within each
modality using learnable queries, thereby improving the discovery of forged
details. Extensive experiments on the $\rm DGM^4$ dataset demonstrate the
superior performance of our proposed model compared to state-of-the-art
approaches.

我们构建了一个简单而新颖的基于 Transformer 的多模态操纵检测和定位任务框架，通过引入视觉 / 语言预训练的编码器和双支路交叉注意力来同时探索模态特定的特征，并提出了解耦的细粒度分类器和自适应地聚合全局上下文线索的隐式操纵查询，从而提高了揭示伪造细节的能力，实验证明我们提出的模型在 $m DGM^4$ 数据集上表现优于现有方法。