Vision-language models (VLMs) offer a promising paradigm for image
classification by comparing the similarity between images and class embeddings.
A critical challenge lies in crafting precise textual representations for class
names. While previous studies have leveraged recent advancements in large
language models (LLMs) to enhance these descriptors, their outputs often suffer
from ambiguity and inaccuracy. We identify two primary causes: 1) The prevalent
reliance on textual interactions with LLMs, leading to a mismatch between the
generated text and the visual content in VLMs' latent space - a phenomenon we
term the "explain without seeing" dilemma. 2) The oversight of the inter-class
relationships, resulting in descriptors that fail to differentiate similar
classes effectively. To address these issues, we propose a novel image
classification framework combining VLMs with LLMs, named Iterative Optimization
with Visual Feedback. In particular, our method develops an LLM-based agent,
employing an evolutionary optimization strategy to refine class descriptors.
Crucially, we incorporate visual feedback from VLM classification metrics,
thereby guiding the optimization process with concrete visual data. Our method
leads to improving accuracy on a wide range of image classification benchmarks,
with 3.47\% average gains over state-of-the-art methods. We also highlight the
resulting descriptions serve as explainable and robust features that can
consistently improve the performance across various backbone models.

将视觉语言模型（VLMs）与大型语言模型（LLMs）相结合的迭代优化与视觉反馈方法，显著提高了图像分类性能，并产生了可解释和稳健的特征描述符。

LLMs 作为视觉解释器：通过演变的视觉描述推进图像分类

LLMs as Visual Explainers: Advancing Image Classification with Evolving  Visual Descriptions

Adversarial training is a useful approach to promote the learning of
transferable representations across the source and target domains, which has
been widely applied for domain adaptation (DA) tasks based on deep neural
networks. Until very recently, existing adversarial domain adaptation (ADA)
methods ignore the useful information from the label space, which is an
important factor accountable for the complicated data distributions associated
with different semantic classes. Especially, the inter-class semantic
relationships have been rarely considered and discussed in the current work of
transfer learning. In this paper, we propose a novel relationship-aware
adversarial domain adaptation (RADA) algorithm, which first utilizes a single
multi-class domain discriminator to enforce the learning of inter-class
dependency structure during domain-adversarial training and then aligns this
structure with the inter-class dependencies that are characterized from
training the label predictor on source domain. Specifically, we impose a
regularization term to penalize the structure discrepancy between the
inter-class dependencies respectively estimated from domain discriminator and
label predictor. Through this alignment, our proposed method makes the
adversarial domain adaptation aware of the class relationships. Empirical
studies show that the incorporation of class relationships significantly
improves the performance on benchmark datasets.

本文提出了一种新的关系感知对抗域自适应（RADA）算法，通过使用一个单一的多类域鉴别器，强制进行领域对抗训练的同时学习 inter-class 依赖关系结构，并将此结构与从源域的标签预测器中表征出的 inter-class 依赖关系对齐，证明了该方法在改善基准数据集性能方面显著提高了类关系的应用。

意识到类关系的对抗域自适应

Adversarial Domain Adaptation Being Aware of Class Relationships

In the context of single-label classification, despite the huge success of
deep learning, the commonly used cross-entropy loss function ignores the
intricate inter-class relationships that often exist in real-life tasks such as
age classification. In this work, we propose to leverage these relationships
between classes by training deep nets with the exact squared Earth Mover's
Distance (also known as Wasserstein distance) for single-label classification.
The squared EMD loss uses the predicted probabilities of all classes and
penalizes the miss-predictions according to a ground distance matrix that
quantifies the dissimilarities between classes. We demonstrate that on datasets
with strong inter-class relationships such as an ordering between classes, our
exact squared EMD losses yield new state-of-the-art results. Furthermore, we
propose a method to automatically learn this matrix using the CNN's own
features during training. We show that our method can learn a ground distance
matrix efficiently with no inter-class relationship priors and yield the same
performance gain. Finally, we show that our method can be generalized to
applications that lack strong inter-class relationships and still maintain
state-of-the-art performance. Therefore, with limited computational overhead,
one can always deploy the proposed loss function on any dataset over the
conventional cross-entropy.

本研究提出将 Wasserstein 距离应用于单标签分类，以考虑现实任务中存在的类别间联系，并表明该方法在具有明显类间关系的数据集上表现出最新的最优结果。