Current approaches for image-based keypoint detection on animal (including
human) body and face are limited to specific keypoints and species. We address
the limitation by proposing the Open-Vocabulary Keypoint Detection (OVKD) task.
It aims to use text prompts to localize arbitrary keypoints of any species. To
accomplish this objective, we propose Open-Vocabulary Keypoint Detection with
Semantic-feature Matching (KDSM), which utilizes both vision and language
models to harness the relationship between text and vision and thus achieve
keypoint detection through associating text prompt with relevant keypoint
features. Additionally, KDSM integrates domain distribution matrix matching and
some special designs to reinforce the relationship between language and vision,
thereby improving the model's generalizability and performance. Extensive
experiments show that our proposed components bring significant performance
improvements, and our overall method achieves impressive results in OVKD.
Remarkably, our method outperforms the state-of-the-art few-shot keypoint
detection methods using a zero-shot fashion. We will make the source code
publicly accessible.

使用图像和语言模型相结合的开放词汇关键点检测方法，通过关联文本提示与相关关键点特征实现任意物种的关键点检测，以实现在零样本情况下超越当前最先进的少样本关键点检测方法，并在 OVKD 上取得了显著的性能改进。

面向动物身体和面部的语言驱动开放词汇关键点检测

Language-driven Open-Vocabulary Keypoint Detection for Animal Body and  Face

Unlike current deep keypoint detectors that are trained to recognize limited
number of body parts, few-shot keypoint detection (FSKD) attempts to localize
any keypoints, including novel or base keypoints, depending on the reference
samples. FSKD requires the semantically meaningful relations for keypoint
similarity learning to overcome the ubiquitous noise and ambiguous local
patterns. One rescue comes with vision transformer (ViT) as it captures
long-range relations well. However, ViT may model irrelevant features outside
of the region of interest due to the global attention matrix, thus degrading
similarity learning between support and query features. In this paper, we
present a novel saliency-guided vision transformer, dubbed SalViT, for few-shot
keypoint detection. Our SalViT enjoys a uniquely designed masked self-attention
and a morphology learner, where the former introduces saliency map as a soft
mask to constrain the self-attention on foregrounds, while the latter leverages
the so-called power normalization to adjust morphology of saliency map,
realizing ``dynamically changing receptive field''. Moreover, as salinecy
detectors add computations, we show that attentive masks of DINO transformer
can replace saliency. On top of SalViT, we also investigate i) transductive
FSKD that enhances keypoint representations with unlabelled data and ii) FSKD
under occlusions. We show that our model performs well on five public datasets
and achieves ~10% PCK higher than the normally trained model under severe
occlusions.

研究提出了一种基于视觉转换器（ViT）的新型显著性引导视觉转换器（SalViT），实现了少样本关键点检测，并通过形态学学习者和软遮罩机制使关键点学习更加精确，同时还为从未见过的关键点和遮挡噪声提供了支持。