This paper proposes a novel framework for multi-label image recognition
without any training data, called data-free framework, which uses knowledge of
pre-trained Large Language Model (LLM) to learn prompts to adapt pretrained
Vision-Language Model (VLM) like CLIP to multilabel classification. Through
asking LLM by well-designed questions, we acquire comprehensive knowledge about
characteristics and contexts of objects, which provides valuable text
descriptions for learning prompts. Then we propose a hierarchical prompt
learning method by taking the multi-label dependency into consideration,
wherein a subset of category-specific prompt tokens are shared when the
corresponding objects exhibit similar attributes or are more likely to
co-occur. Benefiting from the remarkable alignment between visual and
linguistic semantics of CLIP, the hierarchical prompts learned from text
descriptions are applied to perform classification of images during inference.
Our framework presents a new way to explore the synergies between multiple
pre-trained models for novel category recognition. Extensive experiments on
three public datasets (MS-COCO, VOC2007, and NUS-WIDE) demonstrate that our
method achieves better results than the state-of-the-art methods, especially
outperforming the zero-shot multi-label recognition methods by 4.7% in mAP on
MS-COCO.

提出了一种无需训练数据的多标签图像识别新框架，利用预训练大型语言模型（LLM）的知识学习提示，使预训练的视觉 - 语言模型（VLM）如 CLIP 适应多标签分类。通过向 LLM 提问获取关于对象的特性和背景的综合知识，为学习提示提供宝贵的文本描述。然后，通过考虑多标签依赖性，提出了一种层次化的提示学习方法，在对象具有相似属性或更有可能共现时，共享特定类别提示标记的子集。由于 CLIP 在视觉和语义上具有显著的对准性，从文本描述学习到的层次化提示被应用于推理过程中的图像分类。我们的框架为探索多个预训练模型之间的协同作用提供了一种新途径。在三个公共数据集（MS-COCO、VOC2007 和 NUS-WIDE）上进行的大量实验证明，我们的方法比现有方法取得更好的结果，尤其是在 MS-COCO 上的零样本多标签识别方法的 mAP 上超过 4.7%。

基于 LLM 增强的提示调整的无数据多标签图像识别

Data-free Multi-label Image Recognition via LLM-powered Prompt Tuning

In this work, a new data-free framework called Feature Enforcing Physics
Informed Neural Network (FE-PINN) is introduced. This framework is capable of
learning the underlying pattern of any problem with low computational cost
before the main training loop. The loss function of vanilla PINN due to the
existence of two terms of partial differential residuals and boundary condition
mean squared error is imbalanced. FE-PINN solves this challenge with just one
minute of training instead of time-consuming hyperparameter tuning for loss
function that can take hours. The FE-PINN accomplishes this process by
performing a sequence of sub-tasks. The first sub-task learns useful features
about the underlying physics. Then, the model trains on the target task to
refine the calculations. FE-PINN is applied to three benchmarks, flow over a
cylinder, 2D heat conduction, and an inverse problem of calculating inlet
velocity. FE-PINN can solve each case with, 15x, 2x, and 5x speed up
accordingly. Another advantage of FE-PINN is that reaching lower order of value
for loss function is systematically possible. In this study, it was possible to
reach a loss value near 1e-5 which is challenging for vanilla PINN. FE-PINN
also has a smooth convergence process which allows for utilizing higher
learning rates in comparison to vanilla PINN. This framework can be used as a
fast, accurate tool for solving a wide range of Partial Differential Equations
(PDEs) across various fields.

利用 FE-PINN 框架，可以用低计算成本学习任何问题的潜在模式，并解决传统 PINN 框架中的不平衡损失函数的挑战，从而实现快速、准确地解决各领域的偏微分方程问题。