This paper investigates the effectiveness of self-supervised pre-trained
transformers compared to supervised pre-trained transformers and conventional
neural networks (ConvNets) for detecting various types of deepfakes. We focus
on their potential for improved generalization, particularly when training data
is limited. Despite the notable success of large vision-language models
utilizing transformer architectures in various tasks, including zero-shot and
few-shot learning, the deepfake detection community has still shown some
reluctance to adopt pre-trained vision transformers (ViTs), especially large
ones, as feature extractors. One concern is their perceived excessive capacity,
which often demands extensive data, and the resulting suboptimal generalization
when training or fine-tuning data is small or less diverse. This contrasts
poorly with ConvNets, which have already established themselves as robust
feature extractors. Additionally, training and optimizing transformers from
scratch requires significant computational resources, making this accessible
primarily to large companies and hindering broader investigation within the
academic community. Recent advancements in using self-supervised learning (SSL)
in transformers, such as DINO and its derivatives, have showcased significant
adaptability across diverse vision tasks and possess explicit semantic
segmentation capabilities. By leveraging DINO for deepfake detection with
modest training data and implementing partial fine-tuning, we observe
comparable adaptability to the task and the natural explainability of the
detection result via the attention mechanism. Moreover, partial fine-tuning of
transformers for deepfake detection offers a more resource-efficient
alternative, requiring significantly fewer computational resources.

本研究探讨自监督预训练变换器相对于有监督预训练变换器和传统神经网络（ConvNets）在检测各种类型的深度伪造方面的有效性，重点关注其在数据有限的情况下改进泛化能力的潜力。通过使用适度的训练数据并实施部分微调，在利用自监督学习和变换器进行深度伪造检测时，我们观察到了与传统方法可比拟的适应性和通过注意机制实现的自然可解释性，且同时需要较少的计算资源。