Text-video retrieval is an important multi-modal learning task, where the
goal is to retrieve the most relevant video for a given text query. Recently,
pre-trained models, e.g., CLIP, show great potential on this task. However, as
pre-trained models are scaling up, fully fine-tuning them on text-video
retrieval datasets has a high risk of overfitting. Moreover, in practice, it
would be costly to train and store a large model for each task. To overcome the
above issues, we present a novel $\textbf{Cross-Modal Adapter}$ for
parameter-efficient fine-tuning. Inspired by adapter-based methods, we adjust
the pre-trained model with a few parameterization layers. However, there are
two notable differences. First, our method is designed for the multi-modal
domain. Secondly, it allows early cross-modal interactions between CLIP's two
encoders. Although surprisingly simple, our approach has three notable
benefits: (1) reduces $\textbf{99.6}\%$ of fine-tuned parameters, and
alleviates the problem of overfitting, (2) saves approximately 30% of training
time, and (3) allows all the pre-trained parameters to be fixed, enabling the
pre-trained model to be shared across datasets. Extensive experiments
demonstrate that, without bells and whistles, it achieves superior or
comparable performance compared to fully fine-tuned methods on MSR-VTT, MSVD,
VATEX, ActivityNet, and DiDeMo datasets. The code will be available at
https://github.com/LeapLabTHU/Cross-Modal-Adapter.

提出了一种跨模态适配器（Cross-Modal Adapter）方法，对预训练模型进行参数高效微调，可在多模态模型上减少 99.6％的参数、节省 30％的训练时间以及共享预训练模型， 在 MSRVTT、MSVD、VATEX、ActivityNet 和 DiDeMo 数据集上实现了优异或可比的性能。

文本 - 视频检索的跨模态适配器

Cross-Modal Adapter for Text-Video Retrieval

The robustness and generalization ability of Presentation Attack Detection
(PAD) methods is critical to ensure the security of Face Recognition Systems
(FRSs). However, in a real scenario, Presentation Attacks (PAs) are various and
it is hard to predict the Presentation Attack Instrument (PAI) species that
will be used by the attacker. Existing PAD methods are highly dependent on the
limited training set and cannot generalize well to unknown PAI species. Unlike
this specific PAD task, other face related tasks trained by huge amount of real
faces (e.g. face recognition and attribute editing) can be effectively adopted
into different application scenarios. Inspired by this, we propose to trade
position of PAD and face related work in a face system and apply the free
acquired prior knowledge from face related tasks to solve face PAD, so as to
improve the generalization ability in detecting PAs. The proposed method, first
introduces task specific features from other face related task, then, we design
a Cross-Modal Adapter using a Graph Attention Network (GAT) to re-map such
features to adapt to PAD task. Finally, face PAD is achieved by using the
hierarchical features from a CNN-based PA detector and the re-mapped features.
The experimental results show that the proposed method can achieve significant
improvements in the complicated and hybrid datasets, when compared with the
state-of-the-art methods. In particular, when training on the datasets
OULU-NPU, CASIA-FASD, and Idiap Replay-Attack, we obtain HTER (Half Total Error
Rate) of 5.48% for the testing dataset MSU-MFSD, outperforming the baseline by
7.39%.

该论文介绍了一种利用人脸相关任务先验知识来提高 Presentation Attack Detection (PAD) 的泛化能力的方法，该方法包括引入人脸相关任务的特征、设计跨模态适配器以重新映射特征并使用卷积神经网络 (CNN) -basaed PA 检测器的分层特征和重新映射特征来实现 PAD。实验结果表明，该方法在复杂和混合数据集上表现显着优于现有技术。