Although existing semi-supervised learning models achieve remarkable success
in learning with unannotated in-distribution data, they mostly fail to learn on
unlabeled data sampled from novel semantic classes due to their closed-set
assumption. In this work, we target a pragmatic but under-explored Generalized
Novel Category Discovery (GNCD) setting. The GNCD setting aims to categorize
unlabeled training data coming from known and novel classes by leveraging the
information of partially labeled known classes. We propose a two-stage
Contrastive Affinity Learning method with auxiliary visual Prompts, dubbed
PromptCAL, to address this challenging problem. Our approach discovers reliable
pairwise sample affinities to learn better semantic clustering of both known
and novel classes for the class token and visual prompts. First, we propose a
discriminative prompt regularization loss to reinforce semantic
discriminativeness of prompt-adapted pre-trained vision transformer for refined
affinity relationships.Besides, we propose contrastive affinity learning to
calibrate semantic representations based on our iterative semi-supervised
affinity graph generation method for semantically-enhanced supervision.
Extensive experimental evaluation demonstrates that our PromptCAL method is
more effective in discovering novel classes even with limited annotations and
surpasses the current state-of-the-art on generic and fine-grained benchmarks
(e.g., with nearly 11% gain on CUB-200, and 9% on ImageNet-100) on overall
accuracy. Our code is available at this https URL

本文提出了一种两阶段的对比相似度学习方法，名为 PromptCAL，以解决半监督图像分类中的广义新类别发现问题；该方法发现可靠的样本相似性，更好地聚类已知和未知的类别，并在 CUB-200 和 ImageNet-100 等数据集上实现显著效果。

PromptCAL: 通过辅助提示的对比亲和力学习进行广义新类别发现

PromptCAL: Contrastive Affinity Learning via Auxiliary Prompts for  Generalized Novel Category Discovery

This paper is on Few-Shot Object Detection (FSOD), where given a few
templates (examples) depicting a novel class (not seen during training), the
goal is to detect all of its occurrences within a set of images. From a
practical perspective, an FSOD system must fulfil the following desiderata: (a)
it must be used as is, without requiring any fine-tuning at test time, (b) it
must be able to process an arbitrary number of novel objects concurrently while
supporting an arbitrary number of examples from each class and (c) it must
achieve accuracy comparable to a closed system. While there are (relatively)
few systems that support (a), to our knowledge, there is no system supporting
(b) and (c). In this work, we make the following contributions: We introduce,
for the first time, a simple, yet powerful, few-shot detection transformer
(FS-DETR) that can address both desiderata (a) and (b). Our system builds upon
the DETR framework, extending it based on two key ideas: (1) feed the provided
visual templates of the novel classes as visual prompts during test time, and
(2) ``stamp'' these prompts with pseudo-class embeddings, which are then
predicted at the output of the decoder. Importantly, we show that our system is
not only more flexible than existing methods, but also, making a step towards
satisfying desideratum (c), it is more accurate, matching and outperforming the
current state-of-the-art on the most well-established benchmarks (PASCAL VOC &
MSCOCO) for FSOD. Code will be made available.

本文介绍了首个可以解决 Few-Shot Object Detection 中 (a) 和 (b) 两个要求的简单而强大的 Few-Shot Detection Transformer (FS-DETR) 系统，该系统在测试时使用提供的视觉模板作为视觉提示，并使用伪类别嵌入预测。实验结果表明此方法比现有方法更加灵活、准确并达到了当前最先进水平。

FS-DETR: 带提示且无需重新训练的少样本检测变压器

FS-DETR: Few-Shot DEtection TRansformer with prompting and without re-training

Multi-modal tracking gains attention due to its ability to be more accurate
and robust in complex scenarios compared to traditional RGB-based tracking. Its
key lies in how to fuse multi-modal data and reduce the gap between modalities.
However, multi-modal tracking still severely suffers from data deficiency, thus
resulting in the insufficient learning of fusion modules. Instead of building
such a fusion module, in this paper, we provide a new perspective on
multi-modal tracking by attaching importance to the multi-modal visual prompts.
We design a novel multi-modal prompt tracker (ProTrack), which can transfer the
multi-modal inputs to a single modality by the prompt paradigm. By best
employing the tracking ability of pre-trained RGB trackers learning at scale,
our ProTrack can achieve high-performance multi-modal tracking by only altering
the inputs, even without any extra training on multi-modal data. Extensive
experiments on 5 benchmark datasets demonstrate the effectiveness of the
proposed ProTrack.

本研究提出一种基于 prompt 范式的多模态 prompt 跟踪器 ProTrack，通过最大程度地利用 RGB 跟踪器预训练的跟踪能力，仅通过修改输入即可实现高性能的多模态跟踪，对 5 个基准数据集进行的大量实验证明了 ProTrack 的有效性。