Contrastive Language-Image Pre-training (CLIP) has been shown to improve
zero-shot generalization capabilities of language and vision models. In this
paper, we extend CLIP for efficient knowledge distillation, by utilizing
embeddings as teachers. Typical knowledge distillation frameworks require
running forward passes through a teacher model, which is often prohibitive in
the case of billion or trillion parameter teachers. In these cases, using only
the embeddings of the teacher models to guide the distillation can yield
significant computational savings. Our preliminary findings show that
CLIP-based knowledge distillation with embeddings can outperform full scale
knowledge distillation using $9\times$ less memory and $8\times$ less training
time. Code available at: this https URL

通过利用嵌入作为教师，我们扩展了 CLIP 用于高效进行知识蒸馏，初步结果表明，使用嵌入进行基于 CLIP 的知识蒸馏可以在使用少至 9 倍内存和 8 倍训练时间的情况下优于完整规模的知识蒸馏。

CLIP-Embed-KD: 以嵌入向量作为教师的计算高效知识蒸馏

CLIP-Embed-KD: Computationally Efficient Knowledge Distillation Using  Embeddings as Teachers

Existing pre-trained vision-language models, e.g., CLIP, have demonstrated
impressive zero-shot generalization capabilities in various downstream tasks.
However, the performance of these models will degrade significantly when test
inputs present different distributions. To this end, we explore the concept of
test-time prompt tuning (TTPT), which enables the adaptation of the CLIP model
to novel downstream tasks through only one step of optimization on an
unsupervised objective that involves the test sample. Motivated by in-context
learning within field of natural language processing (NLP), we propose
In-Context Prompt Learning (InCPL) for test-time visual recognition task. InCPL
involves associating a new test sample with very few or even just one labeled
example as its in-context prompt. As a result, it can reliably estimate a label
for the test sample, thereby facilitating the model adaptation process. InCPL
first employs a token net to represent language descriptions as visual prompts
that the vision encoder of a CLIP model can comprehend. Paired with in-context
examples, we further propose a context-aware unsupervised loss to optimize test
sample-aware visual prompts. This optimization allows a pre-trained, frozen
CLIP model to be adapted to a test sample from any task using its learned
adaptive prompt. Our method has demonstrated superior performance and achieved
state-of-the-art results across various downstream datasets.

通过测试样本的无监督目标，在视觉识别任务中使用上下文提示学习来适应预训练的视觉 - 语言模型，并取得了在各种下游数据集上的有效结果。

冻结视觉语言模型的测试时视觉识别中的上下文提示学习

In-context Prompt Learning for Test-time Vision Recognition with Frozen  Vision-language Model

We present a learning-based dynamics model for granular material
manipulation. Inspired by the Eulerian approach commonly used in fluid
dynamics, our method adopts a fully convolutional neural network that operates
on a density field-based representation of object piles and pushers, allowing
it to exploit the spatial locality of inter-object interactions as well as the
translation equivariance through convolution operations. Furthermore, our
differentiable action rendering module makes the model fully differentiable and
can be directly integrated with a gradient-based trajectory optimization
algorithm. We evaluate our model with a wide array of piles manipulation tasks
both in simulation and real-world experiments and demonstrate that it
significantly exceeds existing latent or particle-based methods in both
accuracy and computation efficiency, and exhibits zero-shot generalization
capabilities across various environments and tasks.

基于学习的动力学模型用于颗粒材料操纵，通过卷积神经网络实现了对物体堆和推动者的密度场表示，结合可微分的操作渲染模块，与基于梯度的轨迹优化算法进行直接集成，通过在模拟和实验中验证表明其在准确性和计算效率方面明显超越现有的基于潜变量或基于粒子的方法，并在各种环境和任务中展示了零样本泛化能力。