Multi-channel video-language retrieval require models to understand
information from different channels (e.g. video$+$question, video$+$speech) to
correctly link a video with a textual response or query. Fortunately,
contrastive multimodal models are shown to be highly effective at aligning
entities in images/videos and text, e.g., CLIP; text contrastive models are
extensively studied recently for their strong ability of producing
discriminative sentence embeddings, e.g., SimCSE. However, there is not a clear
way to quickly adapt these two lines to multi-channel video-language retrieval
with limited data and resources. In this paper, we identify a principled model
design space with two axes: how to represent videos and how to fuse video and
text information. Based on categorization of recent methods, we investigate the
options of representing videos using continuous feature vectors or discrete
text tokens; for the fusion method, we explore the use of a multimodal
transformer or a pretrained contrastive text model. We extensively evaluate the
four combinations on five video-language datasets. We surprisingly find that
discrete text tokens coupled with a pretrained contrastive text model yields
the best performance, which can even outperform state-of-the-art on the iVQA
and How2QA datasets without additional training on millions of video-text data.
Further analysis shows that this is because representing videos as text tokens
captures the key visual information and text tokens are naturally aligned with
text models that are strong retrievers after the contrastive pretraining
process. All the empirical analysis establishes a solid foundation for future
research on affordable and upgradable multimodal intelligence.

探索多模态检索中利用预训练对比模型和文本符号融合信息的最佳方式，并发现用离散文本符号表示视频的方法取得最佳效果。

用于多通道视频 - 语言检索的预训练对比模型的快速适应

Towards Fast Adaptation of Pretrained Contrastive Models for  Multi-channel Video-Language Retrieval

Deep models trained in supervised mode have achieved remarkable success on a
variety of tasks. When labeled samples are limited, self-supervised learning
(SSL) is emerging as a new paradigm for making use of large amounts of
unlabeled samples. SSL has achieved promising performance on natural language
and image learning tasks. Recently, there is a trend to extend such success to
graph data using graph neural networks (GNNs). In this survey, we provide a
unified review of different ways of training GNNs using SSL. Specifically, we
categorize SSL methods into contrastive and predictive models. In either
category, we provide a unified framework for methods as well as how these
methods differ in each component under the framework. Our unified treatment of
SSL methods for GNNs sheds light on the similarities and differences of various
methods, setting the stage for developing new methods and algorithms. We also
summarize different SSL settings and the corresponding datasets used in each
setting. To facilitate methodological development and empirical comparison, we
develop a standardized testbed for SSL in GNNs, including implementations of
common baseline methods, datasets, and evaluation metrics.

本论文介绍了自监督学习在图神经网络上的应用，将其方法分为对比模型和预测模型并提供统一的框架，在 SSL 方法和数据集上提供了概述和标准化的测试平台。