Contrastive Language-Image Pre-training (CLIP) starts to emerge in many
computer vision tasks and has achieved promising performance. However, it
remains underexplored whether CLIP can be generalized to 3D hand pose
estimation, as bridging text prompts with pose-aware features presents
significant challenges due to the discrete nature of joint positions in 3D
space. In this paper, we make one of the first attempts to propose a novel 3D
hand pose estimator from monocular images, dubbed as CLIP-Hand3D, which
successfully bridges the gap between text prompts and irregular detailed pose
distribution. In particular, the distribution order of hand joints in various
3D space directions is derived from pose labels, forming corresponding text
prompts that are subsequently encoded into text representations.
Simultaneously, 21 hand joints in the 3D space are retrieved, and their spatial
distribution (in x, y, and z axes) is encoded to form pose-aware features.
Subsequently, we maximize semantic consistency for a pair of pose-text features
following a CLIP-based contrastive learning paradigm. Furthermore, a
coarse-to-fine mesh regressor is designed, which is capable of effectively
querying joint-aware cues from the feature pyramid. Extensive experiments on
several public hand benchmarks show that the proposed model attains a
significantly faster inference speed while achieving state-of-the-art
performance compared to methods utilizing the similar scale backbone.

提出了一种称为 CLIP-Hand3D 的新颖的 3D 手势估计器，能够成功地将文本提示与不规则的详细姿势分布连接起来，并通过对姿势标签进行编码，形成相应的文本提示，并从中检索 3D 空间中的 21 个手关节，并对它们的空间分布进行编码以形成姿势感知特征，随后通过构建一个粗到细的网格回归器来最大化姿势 - 文本特征对的语义一致性，实验结果表明该模型相比使用相似规模骨干的方法，在几个公开手部基准测试中实现了更快的推断速度和最先进的性能。