Fine-grained visual classification (FGVC) involves categorizing fine
subdivisions within a broader category, which poses challenges due to subtle
inter-class discrepancies and large intra-class variations. However, prevailing
approaches primarily focus on uni-modal visual concepts. Recent advancements in
pre-trained vision-language models have demonstrated remarkable performance in
various high-level vision tasks, yet the applicability of such models to FGVC
tasks remains uncertain. In this paper, we aim to fully exploit the
capabilities of cross-modal description to tackle FGVC tasks and propose a
novel multimodal prompting solution, denoted as MP-FGVC, based on the
contrastive language-image pertaining (CLIP) model. Our MP-FGVC comprises a
multimodal prompts scheme and a multimodal adaptation scheme. The former
includes Subcategory-specific Vision Prompt (SsVP) and Discrepancy-aware Text
Prompt (DaTP), which explicitly highlights the subcategory-specific
discrepancies from the perspectives of both vision and language. The latter
aligns the vision and text prompting elements in a common semantic space,
facilitating cross-modal collaborative reasoning through a Vision-Language
Fusion Module (VLFM) for further improvement on FGVC. Moreover, we tailor a
two-stage optimization strategy for MP-FGVC to fully leverage the pre-trained
CLIP model and expedite efficient adaptation for FGVC. Extensive experiments
conducted on four FGVC datasets demonstrate the effectiveness of our MP-FGVC.

本文介绍了一种基于 CLIP 模型的多模态提示解决方案（MP-FGVC），通过跨模态描述充分利用 CLIP 模型在细粒度视觉分类（FGVC）任务中的能力，并且通过多模态的协同推理进一步提高 FGVC 的性能。

深入探讨多模式提示在细粒度视觉分类中的应用

Delving into Multimodal Prompting for Fine-grained Visual Classification

Fine-grained visual categorization (FGVC) is challenging due in part to the
fact that it is often difficult to acquire an enough number of training
samples. To employ large models for FGVC without suffering from overfitting,
existing methods usually adopt a strategy of pre-training the models using a
rich set of auxiliary data, followed by fine-tuning on the target FGVC task.
However, the objective of pre-training does not take the target task into
account, and consequently such obtained models are suboptimal for fine-tuning.
To address this issue, we propose in this paper a new deep FGVC model termed
MetaFGNet. Training of MetaFGNet is based on a novel regularized meta-learning
objective, which aims to guide the learning of network parameters so that they
are optimal for adapting to the target FGVC task. Based on MetaFGNet, we also
propose a simple yet effective scheme for selecting more useful samples from
the auxiliary data. Experiments on benchmark FGVC datasets show the efficacy of
our proposed method.

本文提出了一种基于元学习的深度学习模型 MetaFGNet，通过一个新颖的正则化元学习目标来优化网络参数，解决 Fine-grained 视觉分类的训练样本不足问题，并且在辅助数据的筛选上提出了一种简单有效的方案，实验证明了该方法的有效性。