Achieving expressive 3D motion reconstruction and automatic generation for
isolated sign words can be challenging, due to the lack of real-world 3D
sign-word data, the complex nuances of signing motions, and the cross-modal
understanding of sign language semantics. To address these challenges, we
introduce SignAvatar, a framework capable of both word-level sign language
reconstruction and generation. SignAvatar employs a transformer-based
conditional variational autoencoder architecture, effectively establishing
relationships across different semantic modalities. Additionally, this approach
incorporates a curriculum learning strategy to enhance the model's robustness
and generalization, resulting in more realistic motions. Furthermore, we
contribute the ASL3DWord dataset, composed of 3D joint rotation data for the
body, hands, and face, for unique sign words. We demonstrate the effectiveness
of SignAvatar through extensive experiments, showcasing its superior
reconstruction and automatic generation capabilities. The code and dataset are
available on the project page.

通过使用基于变分自编码器的 Transformer 结构和课程学习策略，我们提出了 SignAvatar 框架，在缺乏真实世界中 3D 手语数据、手语动作复杂微妙性和多模态手语语义交叉理解的情况下具备单词级手语重建和生成的能力。我们贡献的 ASL3DWord 数据集包含了身体、手部和面部的 3D 关节旋转数据，通过大量实验展示了 SignAvatar 在重建和自动生成方面的卓越能力。