Predicting future human pose is a fundamental application for machine
intelligence, which drives robots to plan their behavior and paths ahead of
time to seamlessly accomplish human-robot collaboration in real-world 3D
scenarios. Despite encouraging results, existing approaches rarely consider the
effects of the external scene on the motion sequence, leading to pronounced
artifacts and physical implausibilities in the predictions. To address this
limitation, this work introduces a novel multi-modal sense-informed motion
prediction approach, which conditions high-fidelity generation on two modal
information: external 3D scene, and internal human gaze, and is able to
recognize their salience for future human activity. Furthermore, the gaze
information is regarded as the human intention, and combined with both motion
and scene features, we construct a ternary intention-aware attention to
supervise the generation to match where the human wants to reach. Meanwhile, we
introduce semantic coherence-aware attention to explicitly distinguish the
salient point clouds and the underlying ones, to ensure a reasonable
interaction of the generated sequence with the 3D scene. On two real-world
benchmarks, the proposed method achieves state-of-the-art performance both in
3D human pose and trajectory prediction.

未来人体姿势预测是机器智能的基本应用，本研究引入了一种新的多模态感知驱动的运动预测方法，通过结合外部 3D 场景和内部人类凝视信息来实现高保真度的生成，同时考虑人的意图和场景的语义连贯性，该方法在 3D 人体姿势和轨迹预测上取得了最先进的性能。

多模态感知驱动的三维人体动作预测

Multimodal Sense-Informed Prediction of 3D Human Motions

Recent advances with Convolutional Networks (ConvNets) have shifted the
bottleneck for many computer vision tasks to annotated data collection. In this
paper, we present a geometry-driven approach to automatically collect
annotations for human pose prediction tasks. Starting from a generic ConvNet
for 2D human pose, and assuming a multi-view setup, we describe an automatic
way to collect accurate 3D human pose annotations. We capitalize on constraints
offered by the 3D geometry of the camera setup and the 3D structure of the
human body to probabilistically combine per view 2D ConvNet predictions into a
globally optimal 3D pose. This 3D pose is used as the basis for harvesting
annotations. The benefit of the annotations produced automatically with our
approach is demonstrated in two challenging settings: (i) fine-tuning a generic
ConvNet-based 2D pose predictor to capture the discriminative aspects of a
subject's appearance (i.e.,"personalization"), and (ii) training a ConvNet from
scratch for single view 3D human pose prediction without leveraging 3D pose
groundtruth. The proposed multi-view pose estimator achieves state-of-the-art
results on standard benchmarks, demonstrating the effectiveness of our method
in exploiting the available multi-view information.

本文提出了一种基于几何驱动的方法来自动收集人体姿势预测任务的准确的三维人体姿势标注，利用多视角设置、相机设置的 3D 几何和人体身体结构的约束，将每个视图的 2D ConvNet 预测概率组合成全局最优 3D 姿态。