Recent vision-language models are driven by large-scale pretrained models.
However, adapting pretrained models on limited data presents challenges such as
overfitting, catastrophic forgetting, and the cross-modal gap between vision
and language. We introduce a parameter-efficient method to address these
challenges, combining multimodal prompt learning and a transformer-based
mapping network, while keeping the pretrained models frozen. Our experiments on
several video question answering benchmarks demonstrate the superiority of our
approach in terms of performance and parameter efficiency on both zero-shot and
few-shot settings. Our code is available at this https URL

近期的视觉 - 语言模型主要面临有限数据下的过拟合、灾难性遗忘以及视觉与语言之间的跨模态差距等挑战，本研究引入了一种参数高效的方法来解决这些问题，通过结合多模态提示学习和基于 Transformer 的映射网络，在预训练模型的冻结状态下实现。在几个视频问答基准测试中，我们的实验结果证明了我们的方法在性能和参数效率方面在零样本和少样本设置下的优越性。我们的代码可以通过该 https 网址获得。

多模式提示下的零备和少备视频问答

Zero-Shot and Few-Shot Video Question Answering with Multi-Modal Prompts

Knowledge distillation (KD) has been extensively employed to transfer the
knowledge from a large teacher model to the smaller students, where the
parameters of the teacher are fixed (or partially) during training. Recent
studies show that this mode may cause difficulties in knowledge transfer due to
the mismatched model capacities. To alleviate the mismatch problem,
teacher-student joint training methods, e.g., online distillation, have been
proposed, but it always requires expensive computational cost. In this paper,
we present a parameter-efficient and student-friendly knowledge distillation
method, namely PESF-KD, to achieve efficient and sufficient knowledge transfer
by updating relatively few partial parameters. Technically, we first
mathematically formulate the mismatch as the sharpness gap between their
predictive distributions, where we show such a gap can be narrowed with the
appropriate smoothness of the soft label. Then, we introduce an adapter module
for the teacher and only update the adapter to obtain soft labels with
appropriate smoothness. Experiments on a variety of benchmarks show that
PESF-KD can significantly reduce the training cost while obtaining competitive
results compared to advanced online distillation methods. Code will be released
upon acceptance.

本文提出一种参数高效、学生友好的知识蒸馏方法 PESF-KD，通过更新相对较少的参数，实现高效、充分的知识转移，其中引入了适配器模块，将教师的输出转化为合适平滑度的软标签。实验表明，与在线蒸馏方法相比，PESF-KD 能够显著降低培训成本，同时获得有竞争力的结果。