Our work addresses the problem of egocentric human pose estimation from
downwards-facing cameras on head-mounted devices (HMD). This presents a
challenging scenario, as parts of the body often fall outside of the image or
are occluded. Previous solutions minimize this problem by using fish-eye camera
lenses to capture a wider view, but these can present hardware design issues.
They also predict 2D heat-maps per joint and lift them to 3D space to deal with
self-occlusions, but this requires large network architectures which are
impractical to deploy on resource-constrained HMDs. We predict pose from images
captured with conventional rectilinear camera lenses. This resolves hardware
design issues, but means body parts are often out of frame. As such, we
directly regress probabilistic joint rotations represented as matrix Fisher
distributions for a parameterized body model. This allows us to quantify pose
uncertainties and explain out-of-frame or occluded joints. This also removes
the need to compute 2D heat-maps and allows for simplified DNN architectures
which require less compute. Given the lack of egocentric datasets using
rectilinear camera lenses, we introduce the SynthEgo dataset, a synthetic
dataset with 60K stereo images containing high diversity of pose, shape,
clothing and skin tone. Our approach achieves state-of-the-art results for this
challenging configuration, reducing mean per-joint position error by 23%
overall and 58% for the lower body. Our architecture also has eight times fewer
parameters and runs twice as fast as the current state-of-the-art. Experiments
show that training on our synthetic dataset leads to good generalization to
real world images without fine-tuning.

本研究针对佩戴在头部的下视镜头设备上的自我中心人体姿势估计问题进行了探讨。通过使用传统矩形镜头捕捉图像并采用基于矩阵 Fisher 分布的概率性关节旋转回归方法，该方法解决了硬件设计问题并能够量化姿势的不确定性，同时在此具有挑战性的配置中，获得了最先进的结果。

SimpleEgo: 从自主摄像头预测概率性身体姿势

SimpleEgo: Predicting Probabilistic Body Pose from Egocentric Cameras

While head-mounted devices are becoming more compact, they provide egocentric
views with significant self-occlusions of the device user. Hence, existing
methods often fail to accurately estimate complex 3D poses from egocentric
views. In this work, we propose a new transformer-based framework to improve
egocentric stereo 3D human pose estimation, which leverages the scene
information and temporal context of egocentric stereo videos. Specifically, we
utilize 1) depth features from our 3D scene reconstruction module with
uniformly sampled windows of egocentric stereo frames, and 2) human joint
queries enhanced by temporal features of the video inputs. Our method is able
to accurately estimate human poses even in challenging scenarios, such as
crouching and sitting. Furthermore, we introduce two new benchmark datasets,
i.e., UnrealEgo2 and UnrealEgo-RW (RealWorld). The proposed datasets offer a
much larger number of egocentric stereo views with a wider variety of human
motions than the existing datasets, allowing comprehensive evaluation of
existing and upcoming methods. Our extensive experiments show that the proposed
approach significantly outperforms previous methods. We will release
UnrealEgo2, UnrealEgo-RW, and trained models on our project page.

通过使用新的基于 transformer 模型的框架，结合场景信息和长时空上下文，利用头戴设备提供的俯视图来更准确地估计包括蹲下和坐下等复杂动作的三维人体姿态，并提出了两个新的用于综合评估现有和即将推出方法的数据集 UnrealEgo2 和 UnrealEgo-RW。

从主观立体视频中感知 3D 人体姿势

3D Human Pose Perception from Egocentric Stereo Videos

Generating both plausible and accurate full body avatar motion is the key to
the quality of immersive experiences in mixed reality scenarios. Head-Mounted
Devices (HMDs) typically only provide a few input signals, such as head and
hands 6-DoF. Recently, different approaches achieved impressive performance in
generating full body motion given only head and hands signal. However, to the
best of our knowledge, all existing approaches rely on full hand visibility.
While this is the case when, e.g., using motion controllers, a considerable
proportion of mixed reality experiences do not involve motion controllers and
instead rely on egocentric hand tracking. This introduces the challenge of
partial hand visibility owing to the restricted field of view of the HMD. In
this paper, we propose the first unified approach, HMD-NeMo, that addresses
plausible and accurate full body motion generation even when the hands may be
only partially visible. HMD-NeMo is a lightweight neural network that predicts
the full body motion in an online and real-time fashion. At the heart of
HMD-NeMo is the spatio-temporal encoder with novel temporally adaptable mask
tokens that encourage plausible motion in the absence of hand observations. We
perform extensive analysis of the impact of different components in HMD-NeMo
and introduce a new state-of-the-art on AMASS dataset through our evaluation.

在本文中，我们提出了一种名为 HMD-NeMo 的轻量级神经网络，可以在线实时地预测全身动作，解决了只能部分可见手部情况下的全身动作生成问题，并在 AMASS 数据集上达到了新的最先进水平。