We present a novel framework that concurrently tackles hand action
recognition and 3D future hand motion prediction. While previous works focus on
either recognition or prediction, we propose a generative Transformer VAE
architecture to jointly capture both aspects, facilitating realistic motion
prediction by leveraging the short-term hand motion and long-term action
consistency observed across timestamps.To ensure faithful representation of the
semantic dependency and different temporal granularity of hand pose and action,
our framework is decomposed into two cascaded VAE blocks. The lower pose block
models short-span poses, while the upper action block models long-span action.
These are connected by a mid-level feature that represents sub-second series of
hand poses.Our framework is trained across multiple datasets, where pose and
action blocks are trained separately to fully utilize pose-action annotations
of different qualities. Evaluations show that on multiple datasets, the joint
modeling of recognition and prediction improves over separate solutions, and
the semantic and temporal hierarchy enables long-term pose and action modeling.

提出一种新颖的框架，同时处理手势识别和 3D 未来手势预测，借助生成式 Transformer VAE 架构来联合捕捉两个方面，从而实现通过利用时间戳间观察到的短期手势和长期动作一致性，促进现实动作预测，确保手势姿势和动作的语义依赖和不同时间粒度的忠实表示。