3D hand poses are an under-explored modality for action recognition. Poses
are compact yet informative and can greatly benefit applications with limited
compute budgets. However, poses alone offer an incomplete understanding of
actions, as they cannot fully capture objects and environments with which
humans interact. To efficiently model hand-object interactions, we propose
HandFormer, a novel multimodal transformer. HandFormer combines 3D hand poses
at a high temporal resolution for fine-grained motion modeling with sparsely
sampled RGB frames for encoding scene semantics. Observing the unique
characteristics of hand poses, we temporally factorize hand modeling and
represent each joint by its short-term trajectories. This factorized pose
representation combined with sparse RGB samples is remarkably efficient and
achieves high accuracy. Unimodal HandFormer with only hand poses outperforms
existing skeleton-based methods at 5x fewer FLOPs. With RGB, we achieve new
state-of-the-art performance on Assembly101 and H2O with significant
improvements in egocentric action recognition.

手势建模是行为识别的一种未充分探索的方式。我们提出了一种新颖的多模态变换器 HandFormer 来高效地对手势和场景语义进行建模，该方法在处理效率和准确率方面具有显著优势，并在自我中心行为识别方面取得了最新的最好性能。

关于利用三维手势姿势进行动作识别的实用性研究

On the Utility of 3D Hand Poses for Action Recognition

Estimating 3D hand poses from a single RGB image is challenging because depth
ambiguity leads the problem ill-posed. Training hand pose estimators with 3D
hand mesh annotations and multi-view images often results in significant
performance gains. However, existing multi-view datasets are relatively small
with hand joints annotated by off-the-shelf trackers or automated through model
predictions, both of which may be inaccurate and can introduce biases.
Collecting a large-scale multi-view 3D hand pose images with accurate mesh and
joint annotations is valuable but strenuous. In this paper, we design a spin
match algorithm that enables a rigid mesh model matching with any target mesh
ground truth. Based on the match algorithm, we propose an efficient pipeline to
generate a large-scale multi-view hand mesh (MVHM) dataset with accurate 3D
hand mesh and joint labels. We further present a multi-view hand pose
estimation approach to verify that training a hand pose estimator with our
generated dataset greatly enhances the performance. Experimental results show
that our approach achieves the performance of 0.990 in
$\text{AUC}_{\text{20-50}}$ on the MHP dataset compared to the previous
state-of-the-art of 0.939 on this dataset. Our datasset is public available.
\footnote{\url{this https URL}} Our datasset is available
at~\href{https://github.com/Kuzphi/MVHM}{\color{blue}{this https URL}}.

本文提出了一种 spin match 算法，并基于该算法提出了一种有效的多视角手部网格数据集生成方法，证明使用该数据集训练的手部姿态估计器大幅提升性能，从而达到了 0.990 AUC$_{20-50}$ 的性能，且数据集内容公开。

MVHM: 准确三维手部姿态估计的大规模多视图手部网格基准测试

MVHM: A Large-Scale Multi-View Hand Mesh Benchmark for Accurate 3D Hand  Pose Estimation

In this work we study the use of 3D hand poses to recognize first-person
dynamic hand actions interacting with 3D objects. Towards this goal, we
collected RGB-D video sequences comprised of more than 100K frames of 45 daily
hand action categories, involving 26 different objects in several hand
configurations. To obtain hand pose annotations, we used our own mo-cap system
that automatically infers the 3D location of each of the 21 joints of a hand
model via 6 magnetic sensors and inverse kinematics. Additionally, we recorded
the 6D object poses and provide 3D object models for a subset of hand-object
interaction sequences. To the best of our knowledge, this is the first
benchmark that enables the study of first-person hand actions with the use of
3D hand poses. We present an extensive experimental evaluation of RGB-D and
pose-based action recognition by 18 baselines/state-of-the-art approaches. The
impact of using appearance features, poses, and their combinations are
measured, and the different training/testing protocols are evaluated. Finally,
we assess how ready the 3D hand pose estimation field is when hands are
severely occluded by objects in egocentric views and its influence on action
recognition. From the results, we see clear benefits of using hand pose as a
cue for action recognition compared to other data modalities. Our dataset and
experiments can be of interest to communities of 3D hand pose estimation, 6D
object pose, and robotics as well as action recognition.

本文提出了一种基于 3D 手部姿势的第一人称动态手部动作识别方法，并通过收集大量 RGB-D 视频序列对其进行了实验评估。研究结果表明，在进行动作识别时，与其他数据形式相比，手部姿态作为一种线索具有明显的优势。