We study multi-modal few-shot object detection (FSOD) in this paper, using
both few-shot visual examples and class semantic information for detection,
which are complementary to each other by definition. Most of the previous works
on multi-modal FSOD are fine-tuning-based which are inefficient for online
applications. Moreover, these methods usually require expertise like class
names to extract class semantic embedding, which are hard to get for rare
classes. Our approach is motivated by the high-level conceptual similarity of
(metric-based) meta-learning and prompt-based learning to learn generalizable
few-shot and zero-shot object detection models respectively without
fine-tuning. Specifically, we combine the few-shot visual classifier and text
classifier learned via meta-learning and prompt-based learning respectively to
build the multi-modal classifier and detection models. In addition, to fully
exploit the pre-trained language models, we propose meta-learning-based
cross-modal prompting to generate soft prompts for novel classes present in
few-shot visual examples, which are then used to learn the text classifier.
Knowledge distillation is introduced to learn the soft prompt generator without
using human prior knowledge of class names, which may not be available for rare
classes. Our insight is that the few-shot support images naturally include
related context information and semantics of the class. We comprehensively
evaluate the proposed multi-modal FSOD models on multiple few-shot object
detection benchmarks, achieving promising results.

本文介绍了一种基于多模态少样本学习的目标检测方法，使用视觉样本和分类语义信息来检测目标，通过元学习和提示学习相结合，在不需要微调的情况下建立通用少 / 零样本检测模型，同时引入知识蒸馏来解决针对稀有类别缺乏类名称先验知识的问题，通过在多个少样本数据集上的实验来验证该方法的有效性。