Deepfakes are synthetic media generated using deep generative algorithms and
have posed a severe societal and political threat. Apart from facial
manipulation and synthetic voice, recently, a novel kind of deepfakes has
emerged with either audio or visual modalities manipulated. In this regard, a
new generation of multimodal audio-visual deepfake detectors is being
investigated to collectively focus on audio and visual data for multimodal
manipulation detection. Existing multimodal (audio-visual) deepfake detectors
are often based on the fusion of the audio and visual streams from the video.
Existing studies suggest that these multimodal detectors often obtain
equivalent performances with unimodal audio and visual deepfake detectors. We
conjecture that the heterogeneous nature of the audio and visual signals
creates distributional modality gaps and poses a significant challenge to
effective fusion and efficient performance. In this paper, we tackle the
problem at the representation level to aid the fusion of audio and visual
streams for multimodal deepfake detection. Specifically, we propose the joint
use of modality (audio and visual) invariant and specific representations. This
ensures that the common patterns and patterns specific to each modality
representing pristine or fake content are preserved and fused for multimodal
deepfake manipulation detection. Our experimental results on FakeAVCeleb and
KoDF audio-visual deepfake datasets suggest the enhanced accuracy of our
proposed method over SOTA unimodal and multimodal audio-visual deepfake
detectors by $17.8$% and $18.4$%, respectively. Thus, obtaining
state-of-the-art performance.

本文研究针对深度伪造技术的威胁，提出了一种在表征层面上辅助音频和视觉融合的多模态深度伪造检测方法，通过使用模态不变和特定的表 征确保保留表示真实或伪造内容的共同模式和每个模态的特定模式，实验证明该方法相比于单模态和多模态音 视频深度伪造检测的最先进方法能够提高 17.8% 和 18.4% 的准确性。