Large-scale Pre-Training Vision-Language Model such as CLIP has demonstrated
outstanding performance in zero-shot classification, e.g. achieving 76.3% top-1
accuracy on ImageNet without seeing any example, which leads to potential
benefits to many tasks that have no labeled data. However, while applying CLIP
to a downstream target domain, the presence of visual and text domain gaps and
cross-modality misalignment can greatly impact the model performance. To
address such challenges, we propose ReCLIP, the first source-free domain
adaptation method for vision-language models, which does not require any source
data or target labeled data. ReCLIP first learns a projection space to mitigate
the misaligned visual-text embeddings and learns pseudo labels, and then
deploys cross-modality self-training with the pseudo labels, to update visual
and text encoders, refine labels and reduce domain gaps and misalignments
iteratively. With extensive experiments, we demonstrate ReCLIP reduces the
average error rate of CLIP from 30.17% to 25.06% on 22 image classification
benchmarks.

通过提出 ReCLIP 方法，第一个无需源数据或目标标记数据的视觉 - 语言模型领域自适应方法，该方法使用伪标签进行交叉模态自训练以减小领域间差异和错位对模型性能的影响，在 22 个图像分类基准测试中将 CLIP 的平均错误率从 30.17% 降低到 25.06%。

ReCLIP：使用源域自适应提炼对比语言图像预训练

ReCLIP: Refine Contrastive Language Image Pre-Training with Source Free  Domain Adaptation

Training a referring expression comprehension (ReC) model for a new visual
domain requires collecting referring expressions, and potentially corresponding
bounding boxes, for images in the domain. While large-scale pre-trained models
are useful for image classification across domains, it remains unclear if they
can be applied in a zero-shot manner to more complex tasks like ReC. We present
ReCLIP, a simple but strong zero-shot baseline that repurposes CLIP, a
state-of-the-art large-scale model, for ReC. Motivated by the close connection
between ReC and CLIP's contrastive pre-training objective, the first component
of ReCLIP is a region-scoring method that isolates object proposals via
cropping and blurring, and passes them to CLIP. However, through controlled
experiments on a synthetic dataset, we find that CLIP is largely incapable of
performing spatial reasoning off-the-shelf. Thus, the second component of
ReCLIP is a spatial relation resolver that handles several types of spatial
relations. We reduce the gap between zero-shot baselines from prior work and
supervised models by as much as 29% on RefCOCOg, and on RefGTA (video game
imagery), ReCLIP's relative improvement over supervised ReC models trained on
real images is 8%.

本文介绍了一种基于大规模预训练模型 CLIP 的零 - shot 基线模型 ReCLIP，包含提取物体区域和处理空间关系两个组成部分，实验结果表明相较于其他基线模型， ReCLIP 在指代理解任务上对真实图像的效果提升了 8%。