Vision-Language Models (VLMs) such as CLIP are trained on large amounts of
image-text pairs, resulting in remarkable generalization across several data
distributions. The prohibitively expensive training and data
collection/curation costs of these models make them valuable Intellectual
Property (IP) for organizations. This motivates a vendor-client paradigm, where
a vendor trains a large-scale VLM and grants only input-output access to
clients on a pay-per-query basis in a black-box setting. The client aims to
minimize inference cost by distilling the VLM to a student model using the
limited available task-specific data, and further deploying this student model
in the downstream application. While naive distillation largely improves the
In-Domain (ID) accuracy of the student, it fails to transfer the superior
out-of-distribution (OOD) generalization of the VLM teacher using the limited
available labeled images. To mitigate this, we propose Vision-Language to
Vision-Align, Distill, Predict (VL2V-ADiP), which first aligns the vision and
language modalities of the teacher model with the vision modality of a
pre-trained student model, and further distills the aligned VLM embeddings to
the student. This maximally retains the pre-trained features of the student,
while also incorporating the rich representations of the VLM image encoder and
the superior generalization of the text embeddings. The proposed approach
achieves state-of-the-art results on the standard Domain Generalization
benchmarks in a black-box teacher setting, and also when weights of the VLM are
accessible.

Vision-Language to Vision-Align, Distill, Predict (VL2V-ADiP) is a proposed approach that aligns vision and language modalities to distill pre-trained features and superior generalization for state-of-the-art results in Domain Generalization using Vision-Language Models like CLIP.