Learning to recognize novel concepts from just a few image samples is very
challenging as the learned model is easily overfitted on the few data and
results in poor generalizability. One promising but underexplored solution is
to compensate the novel classes by generating plausible samples. However, most
existing works of this line exploit visual information only, rendering the
generated data easy to be distracted by some challenging factors contained in
the few available samples. Being aware of the semantic information in the
textual modality that reflects human concepts, this work proposes a novel
framework that exploits semantic relations to guide dual-view data
hallucination for few-shot image recognition. The proposed framework enables
generating more diverse and reasonable data samples for novel classes through
effective information transfer from base classes. Specifically, an
instance-view data hallucination module hallucinates each sample of a novel
class to generate new data by employing local semantic correlated attention and
global semantic feature fusion derived from base classes. Meanwhile, a
prototype-view data hallucination module exploits semantic-aware measure to
estimate the prototype of a novel class and the associated distribution from
the few samples, which thereby harvests the prototype as a more stable sample
and enables resampling a large number of samples. We conduct extensive
experiments and comparisons with state-of-the-art methods on several popular
few-shot benchmarks to verify the effectiveness of the proposed framework.

通过利用语义关系引导双视图数据幻影，为小样本图像识别提供更多样化和合理化的新数据样本，该框架能通过从基础类别中进行有效的信息传递生成新颖类别的样本。其中，实例视图数据幻影模块利用基础类别的局部语义相关注意力和全局语义特征融合生成新颖类别的每个样本；原型视图数据幻影模块利用语义感知度量来估计新颖类别的原型和相关分布，从而实现对大量样本的重新采样，提高样本的稳定性。通过在几个常用的小样本数据集上与最先进的方法进行了广泛的实验和比较，验证了该框架的有效性。

基于语义关系引导的双视角数据超分辨率方法用于少样本图像识别

Dual-View Data Hallucination with Semantic Relation Guidance for  Few-Shot Image Recognition

Vision-language models have recently shown great potential on many computer
vision tasks. Meanwhile, prior work demonstrates prompt tuning designed for
vision-language models could acquire superior performance on few-shot image
recognition compared to linear probe, a strong baseline. In real-world
applications, many few-shot tasks are correlated, particularly in a specialized
area. However, such information is ignored by previous work. Inspired by the
fact that modeling task relationships by multi-task learning can usually boost
performance, we propose a novel method SoftCPT (Soft Context Sharing for Prompt
Tuning) to fine-tune pre-trained vision-language models on multiple target
few-shot tasks, simultaneously. Specifically, we design a task-shared meta
network to generate prompt vector for each task using pre-defined task name
together with a learnable meta prompt as input. As such, the prompt vectors of
all tasks will be shared in a soft manner. The parameters of this shared meta
network as well as the meta prompt vector are tuned on the joint training set
of all target tasks. Extensive experiments on three multi-task few-shot
datasets show that SoftCPT outperforms the representative single-task prompt
tuning method CoOp [78] by a large margin, implying the effectiveness of
multi-task learning in vision-language prompt tuning. The source code and data
will be made publicly available.

本研究提出了一种基于多任务学习的视觉语言模型 Prompt Tuning 的方法 SoftCPT，用于同时优化多个 Few-Shot 任务的任务共享元网络，结果表明该模型能够通过软共享模式下的多任务学习实现比 CoOp 更好的结果。