Achieving expressive 3D motion reconstruction and automatic generation for
isolated sign words can be challenging, due to the lack of real-world 3D
sign-word data, the complex nuances of signing motions, and the cross-modal
understanding of sign language semantics. To address these challenges, we
introduce SignAvatar, a framework capable of both word-level sign language
reconstruction and generation. SignAvatar employs a transformer-based
conditional variational autoencoder architecture, effectively establishing
relationships across different semantic modalities. Additionally, this approach
incorporates a curriculum learning strategy to enhance the model's robustness
and generalization, resulting in more realistic motions. Furthermore, we
contribute the ASL3DWord dataset, composed of 3D joint rotation data for the
body, hands, and face, for unique sign words. We demonstrate the effectiveness
of SignAvatar through extensive experiments, showcasing its superior
reconstruction and automatic generation capabilities. The code and dataset are
available on the project page.

通过使用基于变分自编码器的 Transformer 结构和课程学习策略，我们提出了 SignAvatar 框架，在缺乏真实世界中 3D 手语数据、手语动作复杂微妙性和多模态手语语义交叉理解的情况下具备单词级手语重建和生成的能力。我们贡献的 ASL3DWord 数据集包含了身体、手部和面部的 3D 关节旋转数据，通过大量实验展示了 SignAvatar 在重建和自动生成方面的卓越能力。

双立体：手语三维动作重建与生成

SignAvatar: Sign Language 3D Motion Reconstruction and Generation

In this paper, we introduce a method to automatically reconstruct the 3D
motion of a person interacting with an object from a single RGB video. Our
method estimates the 3D poses of the person and the object, contact positions,
and forces and torques actuated by the human limbs. The main contributions of
this work are three-fold. First, we introduce an approach to jointly estimate
the motion and the actuation forces of the person on the manipulated object by
modeling contacts and the dynamics of their interactions. This is cast as a
large-scale trajectory optimization problem. Second, we develop a method to
automatically recognize from the input video the position and timing of
contacts between the person and the object or the ground, thereby significantly
simplifying the complexity of the optimization. Third, we validate our approach
on a recent MoCap dataset with ground truth contact forces and demonstrate its
performance on a new dataset of Internet videos showing people manipulating a
variety of tools in unconstrained environments.

本文提出了一种从单个 RGB 视频自动重建与对象的人交互的 3D 运动的方法，估计人和物体的 3D 姿势，接触位置，被人类肢体激活的力和扭矩，方法主要集中在联合估计运动和动作力，运用大规模的轨迹优化问题，并且从输入视频自动识别出人与物体或地面之间的接触位置和时间，于真实数据集上验证了方法并展示了它在新的互联网视频数据集中的性能。