Deepfake technologies empowered by deep learning are rapidly evolving,
creating new security concerns for society. Existing multimodal detection
methods usually capture audio-visual inconsistencies to expose Deepfake videos.
More seriously, the advanced Deepfake technology realizes the audio-visual
calibration of the critical phoneme-viseme regions, achieving a more realistic
tampering effect, which brings new challenges. To address this problem, we
propose a novel Deepfake detection method to mine the correlation between
Non-critical Phonemes and Visemes, termed NPVForensics. Firstly, we propose the
Local Feature Aggregation block with Swin Transformer (LFA-ST) to construct
non-critical phoneme-viseme and corresponding facial feature streams
effectively. Secondly, we design a loss function for the fine-grained motion of
the talking face to measure the evolutionary consistency of non-critical
phoneme-viseme. Next, we design a phoneme-viseme awareness module for
cross-modal feature fusion and representation alignment, so that the modality
gap can be reduced and the intrinsic complementarity of the two modalities can
be better explored. Finally, a self-supervised pre-training strategy is
leveraged to thoroughly learn the audio-visual correspondences in natural
videos. In this manner, our model can be easily adapted to the downstream
Deepfake datasets with fine-tuning. Extensive experiments on existing
benchmarks demonstrate that the proposed approach outperforms state-of-the-art
methods.

本文提出了一种新的 Deepfake 检测方法（称为 NPVForensics），在现有的多模态检测方法基础上，通过挖掘非关键音位和唇位相关性，构建相应面部特征流，设计精细的动作损失函数，以及跨模态特征融合和表示对齐模块，最终使用自监督预训练策略，在现有基准测试中取得了更优的检测效果。