While 3D human body modeling has received much attention in computer vision,
modeling the acoustic equivalent, i.e. modeling 3D spatial audio produced by
body motion and speech, has fallen short in the community. To close this gap,
we present a model that can generate accurate 3D spatial audio for full human
bodies. The system consumes, as input, audio signals from headset microphones
and body pose, and produces, as output, a 3D sound field surrounding the
transmitter's body, from which spatial audio can be rendered at any arbitrary
position in the 3D space. We collect a first-of-its-kind multimodal dataset of
human bodies, recorded with multiple cameras and a spherical array of 345
microphones. In an empirical evaluation, we demonstrate that our model can
produce accurate body-induced sound fields when trained with a suitable loss.
Dataset and code are available online.

通过从头戴式麦克风获取的音频信号和身体姿势作为输入，生成围绕发射者身体的三维声场，从而可以在三维空间的任意位置呈现空间音频。

身体声响：利用身体姿势和音频进行人体 3D 空间声音建模

Sounding Bodies: Modeling 3D Spatial Sound of Humans Using Body Pose and  Audio

Understanding social interactions from egocentric views is crucial for many
applications, ranging from assistive robotics to AR/VR. Key to reasoning about
interactions is to understand the body pose and motion of the interaction
partner from the egocentric view. However, research in this area is severely
hindered by the lack of datasets. Existing datasets are limited in terms of
either size, capture/annotation modalities, ground-truth quality, or
interaction diversity. We fill this gap by proposing EgoBody, a novel
large-scale dataset for human pose, shape and motion estimation from egocentric
views, during interactions in complex 3D scenes. We employ Microsoft HoloLens2
headsets to record rich egocentric data streams (including RGB, depth, eye
gaze, head and hand tracking). To obtain accurate 3D ground truth, we calibrate
the headset with a multi-Kinect rig and fit expressive SMPL-X body meshes to
multi-view RGB-D frames, reconstructing 3D human shapes and poses relative to
the scene, over time. We collect 125 sequences, spanning diverse interaction
scenarios, and propose the first benchmark for 3D full-body pose and shape
estimation of the social partner from egocentric views. We extensively evaluate
state-of-the-art methods, highlight their limitations in the egocentric
scenario, and address such limitations leveraging our high-quality annotations.
Data and code are available at
this https URL

EgoBody 是一个新的大规模数据集，用于通过自我主观视角获取人类姿态、形状和运动的估计，数据集覆盖丰富的互动场景，估计社交伙伴的 3D 全身姿态和形状。