Pre-trained vision-language models like CLIP have shown powerful zero-shot
inference ability via image-text matching and prove to be strong few-shot
learners in various downstream tasks. However, in real-world scenarios,
adapting CLIP to downstream tasks may encounter the following challenges: 1)
data may exhibit long-tailed data distributions and might not have abundant
samples for all the classes; 2) There might be emerging tasks with new classes
that contain no samples at all. To overcome them, we propose a novel framework
to achieve efficient and long-tailed generalization, which can be termed as
Candle. During the training process, we propose compensating logit-adjusted
loss to encourage large margins of prototypes and alleviate imbalance both
within the base classes and between the base and new classes. For efficient
adaptation, we treat the CLIP model as a black box and leverage the extracted
features to obtain visual and textual prototypes for prediction. To make full
use of multi-modal information, we also propose cross-modal attention to enrich
the features from both modalities. For effective generalization, we introduce
virtual prototypes for new classes to make up for their lack of training
images. Candle achieves state-of-the-art performance over extensive experiments
on 11 diverse datasets while substantially reducing the training time,
demonstrating the superiority of our approach. The source code is available at
this https URL

针对使 CLIP 适应现实世界的挑战，我们提出了一种名为 Candle 的新框架，通过引入新的损失函数、跨模态注意力和虚拟原型来实现高效、长尾泛化，该方法在 11 个不同数据集上展示出了卓越的性能，并大大减少了训练时间。

预训练视觉 - 语言模型的高效和长尾泛化

Efficient and Long-Tailed Generalization for Pre-trained Vision-Language  Model

The sequential process of conceptualization and instantiation is essential to
generalizable commonsense reasoning as it allows the application of existing
knowledge to unfamiliar scenarios. However, existing works tend to undervalue
the step of instantiation and heavily rely on pre-built concept taxonomies and
human annotations to collect both types of knowledge, resulting in a lack of
instantiated knowledge to complete reasoning, high cost, and limited
scalability. To tackle these challenges, we introduce CANDLE, a distillation
framework that iteratively performs contextualized conceptualization and
instantiation over commonsense knowledge bases by instructing large language
models to generate both types of knowledge with critic filtering. By applying
CANDLE to ATOMIC, we construct a comprehensive knowledge base comprising six
million conceptualizations and instantiated commonsense knowledge triples. Both
types of knowledge are firmly rooted in the original ATOMIC dataset, and
intrinsic evaluations demonstrate their exceptional quality and diversity.
Empirical results indicate that distilling CANDLE on student models provides
benefits across four downstream tasks. Our code, data, and models are publicly
available at this https URL

通过引入 CANDLE，一个渗透式谐谑缠绕呼出器，研究人员在常识知识库上进行了环境相关的概念化和实例化，并通过评估验证提出的这些知识类型的杰出质量和多样性，进而得出结论，在学生模型上提取 CANDLE 可以对四个下游任务带来好处。

CANDLE: 大型语言模型中的常识推理的迭代概念化与实例提炼

CANDLE: Iterative Conceptualization and Instantiation Distillation from  Large Language Models for Commonsense Reasoning

Structured knowledge is important for many AI applications. Commonsense
knowledge, which is crucial for robust human-centric AI, is covered by a small
number of structured knowledge projects. However, they lack knowledge about
human traits and behaviors conditioned on socio-cultural contexts, which is
crucial for situative AI. This paper presents CANDLE, an end-to-end methodology
for extracting high-quality cultural commonsense knowledge (CCSK) at scale.
CANDLE extracts CCSK assertions from a huge web corpus and organizes them into
coherent clusters, for 3 domains of subjects (geography, religion, occupation)
and several cultural facets (food, drinks, clothing, traditions, rituals,
behaviors). CANDLE includes judicious techniques for classification-based
filtering and scoring of interestingness. Experimental evaluations show the
superiority of the CANDLE CCSK collection over prior works, and an extrinsic
use case demonstrates the benefits of CCSK for the GPT-3 language model. Code
and data can be accessed at this https URL.

本文提出了一个用于在大规模上提取高质量文化常识知识 (CCsk) 的端到端方法，名为 CANDLE，该方法从 Web 语料库中提取 CCsk 断言并将它们组织成相符的类群，包括地理、宗教、职业等 3 个领域，多个文化方面。通过分类过滤和趣味分数评估方法，CANDLE 优于先前的作品，并且实验评估演示了 CCsk 对 GPT-3 语言模型带来的好处。