Recently, zero-shot multi-label classification has garnered considerable
attention for its capacity to operate predictions on unseen labels without
human annotations. Nevertheless, prevailing approaches often use seen classes
as imperfect proxies for unseen ones, resulting in suboptimal performance.
Drawing inspiration from the success of text-to-image generation models in
producing realistic images, we propose an innovative solution: generating
synthetic data to construct a training set explicitly tailored for proxyless
training on unseen labels. Our approach introduces a novel image generation
framework that produces multi-label synthetic images of unseen classes for
classifier training. To enhance diversity in the generated images, we leverage
a pre-trained large language model to generate diverse prompts. Employing a
pre-trained multi-modal CLIP model as a discriminator, we assess whether the
generated images accurately represent the target classes. This enables
automatic filtering of inaccurately generated images, preserving classifier
accuracy. To refine text prompts for more precise and effective multi-label
object generation, we introduce a CLIP score-based discriminative loss to
fine-tune the text encoder in the diffusion model. Additionally, to enhance
visual features on the target task while maintaining the generalization of
original features and mitigating catastrophic forgetting resulting from
fine-tuning the entire visual encoder, we propose a feature fusion module
inspired by transformer attention mechanisms. This module aids in capturing
global dependencies between multiple objects more effectively. Extensive
experimental results validate the effectiveness of our approach, demonstrating
significant improvements over state-of-the-art methods.

通过生成合成数据，用于训练未见标签的分类器，本文介绍了一种用于零样本多标签分类的创新解决方案，包括使用大型语言模型生成多样的提示，使用 CLIP 模型进行图像筛选，以及使用特征融合模块来提高目标任务的视觉特征。实验结果验证了该方法的有效性，并显示出与现有方法相比的显著改进。

多样化和个性化的图像生成用于零 - shot 多标签分类

Diverse and Tailored Image Generation for Zero-shot Multi-label  Classification

The pandemic resulted in vast repositories of unstructured data, including
radiology reports, due to increased medical examinations. Previous research on
automated diagnosis of COVID-19 primarily focuses on X-ray images, despite
their lower precision compared to computed tomography (CT) scans. In this work,
we leverage unstructured data from a hospital and harness the fine-grained
details offered by CT scans to perform zero-shot multi-label classification
based on contrastive visual language learning. In collaboration with human
experts, we investigate the effectiveness of multiple zero-shot models that aid
radiologists in detecting pulmonary embolisms and identifying intricate lung
details like ground glass opacities and consolidations. Our empirical analysis
provides an overview of the possible solutions to target such fine-grained
tasks, so far overlooked in the medical multimodal pretraining literature. Our
investigation promises future advancements in the medical image analysis
community by addressing some challenges associated with unstructured data and
fine-grained multi-label classification.

使用未结构化数据和 CT 扫描进行零样本多标签分类来诊断 COVID-19，以改善医学影像分析中的精度和效果。