This work proposes a unified framework called UniPose to detect keypoints of
any articulated (e.g., human and animal), rigid, and soft objects via visual or
textual prompts for fine-grained vision understanding and manipulation.
Keypoint is a structure-aware, pixel-level, and compact representation of any
object, especially articulated objects. Existing fine-grained promptable tasks
mainly focus on object instance detection and segmentation but often fail to
identify fine-grained granularity and structured information of image and
instance, such as eyes, leg, paw, etc. Meanwhile, prompt-based keypoint
detection is still under-explored. To bridge the gap, we make the first attempt
to develop an end-to-end prompt-based keypoint detection framework called
UniPose to detect keypoints of any objects. As keypoint detection tasks are
unified in this framework, we can leverage 13 keypoint detection datasets with
338 keypoints across 1,237 categories over 400K instances to train a generic
keypoint detection model. UniPose can effectively align text-to-keypoint and
image-to-keypoint due to the mutual enhancement of textual and visual prompts
based on the cross-modality contrastive learning optimization objectives. Our
experimental results show that UniPose has strong fine-grained localization and
generalization abilities across image styles, categories, and poses. Based on
UniPose as a generalist keypoint detector, we hope it could serve fine-grained
visual perception, understanding, and generation.

该研究提出了一个名为 UniPose 的统一框架，通过视觉或文本提示实现对人体、动物、刚性和柔性物体的关键点检测，以实现细粒度视觉理解和操作。