Predicting future human pose is a fundamental application for machine
intelligence, which drives robots to plan their behavior and paths ahead of
time to seamlessly accomplish human-robot collaboration in real-world 3D
scenarios. Despite encouraging results, existing approaches rarely consider the
effects of the external scene on the motion sequence, leading to pronounced
artifacts and physical implausibilities in the predictions. To address this
limitation, this work introduces a novel multi-modal sense-informed motion
prediction approach, which conditions high-fidelity generation on two modal
information: external 3D scene, and internal human gaze, and is able to
recognize their salience for future human activity. Furthermore, the gaze
information is regarded as the human intention, and combined with both motion
and scene features, we construct a ternary intention-aware attention to
supervise the generation to match where the human wants to reach. Meanwhile, we
introduce semantic coherence-aware attention to explicitly distinguish the
salient point clouds and the underlying ones, to ensure a reasonable
interaction of the generated sequence with the 3D scene. On two real-world
benchmarks, the proposed method achieves state-of-the-art performance both in
3D human pose and trajectory prediction.

未来人体姿势预测是机器智能的基本应用，本研究引入了一种新的多模态感知驱动的运动预测方法，通过结合外部 3D 场景和内部人类凝视信息来实现高保真度的生成，同时考虑人的意图和场景的语义连贯性，该方法在 3D 人体姿势和轨迹预测上取得了最先进的性能。