Weakly supervised vision-and-language pre-training (WVLP), which learns
cross-modal representations with limited cross-modal supervision, has been
shown to effectively reduce the data cost of pre-training while maintaining
decent performance on downstream tasks. However, current WVLP methods use only
local descriptions of images, i.e., object tags, as cross-modal anchors to
construct weakly-aligned image-text pairs for pre-training. This affects the
data quality and thus the effectiveness of pre-training. In this paper, we
propose to directly take a small number of aligned image-text pairs as anchors,
and represent each unaligned image and text by its similarities to these
anchors, i.e., relative representations. We build a WVLP framework based on the
relative representations, namely RELIT, which collects high-quality
weakly-aligned image-text pairs from large-scale image-only and text-only data
for pre-training through relative representation-based retrieval and
generation. Experiments on four downstream tasks show that RELIT achieves new
state-of-the-art results under the weakly supervised setting.

本研究提出了使用相对表示（relative representations）来构建跨模态锚点，从而实现高质量的图像 - 文本对预训练的 WVLP 框架 RELIT，并通过四个下游任务的实验表现得出了新的最先进结果。

基于相对表示的弱监督视觉语言预训练

Weakly Supervised Vision-and-Language Pre-training with Relative  Representations

The use of relative representations for latent embeddings has shown potential
in enabling latent space communication and zero-shot model stitching across a
wide range of applications. Nevertheless, relative representations rely on a
certain amount of parallel anchors to be given as input, which can be
impractical to obtain in certain scenarios. To overcome this limitation, we
propose an optimization-based method to discover new parallel anchors from a
limited number of seeds. Our approach can be used to find semantic
correspondence between different domains, align their relative spaces, and
achieve competitive results in several tasks.

本文提出了一种优化方法，以从有限数量的种子中发现新的并行锚点，用于发现不同领域之间的语义对应关系，对齐它们的相对空间，并在多个任务中实现具有竞争力的结果，这展示了相对表示在启用潜在空间通信和零样本模型缝合方面具有潜力。