Modeling and rendering photorealistic avatars is of crucial importance in
many applications. Existing methods that build a 3D avatar from visual
observations, however, struggle to reconstruct clothed humans. We introduce
PhysAvatar, a novel framework that combines inverse rendering with inverse
physics to automatically estimate the shape and appearance of a human from
multi-view video data along with the physical parameters of the fabric of their
clothes. For this purpose, we adopt a mesh-aligned 4D Gaussian technique for
spatio-temporal mesh tracking as well as a physically based inverse renderer to
estimate the intrinsic material properties. PhysAvatar integrates a physics
simulator to estimate the physical parameters of the garments using
gradient-based optimization in a principled manner. These novel capabilities
enable PhysAvatar to create high-quality novel-view renderings of avatars
dressed in loose-fitting clothes under motions and lighting conditions not seen
in the training data. This marks a significant advancement towards modeling
photorealistic digital humans using physically based inverse rendering with
physics in the loop. Our project website is at:
this https URL

采用逆渲染和逆物理学相结合的方法，PhysAvatar 能够建立逼真的数字人物模型，特别是穿着松散的衣物并能在未见过的动作和光照条件下生成高质量的新视角渲染效果。

PhysAvatar：从视觉观察中学习装束的 3D 化身的物理学

PhysAvatar: Learning the Physics of Dressed 3D Avatars from Visual  Observations

Real-time and robust photorealistic avatars for telepresence in AR/VR have
been highly desired for enabling immersive photorealistic telepresence.
However, there still exists one key bottleneck: the considerable computational
expense needed to accurately infer facial expressions captured from
headset-mounted cameras with a quality level that can match the realism of the
avatar's human appearance. To this end, we propose a framework called
Auto-CARD, which for the first time enables real-time and robust driving of
Codec Avatars when exclusively using merely on-device computing resources. This
is achieved by minimizing two sources of redundancy. First, we develop a
dedicated neural architecture search technique called AVE-NAS for avatar
encoding in AR/VR, which explicitly boosts both the searched architectures'
robustness in the presence of extreme facial expressions and hardware
friendliness on fast evolving AR/VR headsets. Second, we leverage the temporal
redundancy in consecutively captured images during continuous rendering and
develop a mechanism dubbed LATEX to skip the computation of redundant frames.
Specifically, we first identify an opportunity from the linearity of the latent
space derived by the avatar decoder and then propose to perform adaptive latent
extrapolation for redundant frames. For evaluation, we demonstrate the efficacy
of our Auto-CARD framework in real-time Codec Avatar driving settings, where we
achieve a 5.05x speed-up on Meta Quest 2 while maintaining a comparable or even
better animation quality than state-of-the-art avatar encoder designs.

本文提出了一种名为 Auto-CARD 的框架，该框架通过最小化冗余源来启用仅使用设备计算资源时实时和鲁棒的 Codec 头像驱动，实现了在虚拟现实 / 增强现实中建立逼真的远程存在的可能性。

自动 CARD: 面向实时移动遥控的高效稳健编解码头像驱动

Auto-CARD: Efficient and Robust Codec Avatar Driving for Real-time  Mobile Telepresence

Photorealistic avatars of human faces have come a long way in recent years,
yet research along this area is limited by a lack of publicly available,
high-quality datasets covering both, dense multi-view camera captures, and rich
facial expressions of the captured subjects. In this work, we present
Multiface, a new multi-view, high-resolution human face dataset collected from
13 identities at Reality Labs Research for neural face rendering. We introduce
Mugsy, a large scale multi-camera apparatus to capture high-resolution
synchronized videos of a facial performance. The goal of Multiface is to close
the gap in accessibility to high quality data in the academic community and to
enable research in VR telepresence. Along with the release of the dataset, we
conduct ablation studies on the influence of different model architectures
toward the model's interpolation capacity of novel viewpoint and expressions.
With a conditional VAE model serving as our baseline, we found that adding
spatial bias, texture warp field, and residual connections improves performance
on novel view synthesis. Our code and data is available at:
this https URL

该研究介绍了一个多视角，高分辨率的人脸数据集，通过进行替代卷积编码器及对条件对抗网络（VAE）进行验证，对研究区域的深度插值进行了实验讨论。

Multiface: 神经人脸渲染数据集

Multiface: A Dataset for Neural Face Rendering

Telecommunication with photorealistic avatars in virtual or augmented reality
is a promising path for achieving authentic face-to-face communication in 3D
over remote physical distances. In this work, we present the Pixel Codec
Avatars (PiCA): a deep generative model of 3D human faces that achieves state
of the art reconstruction performance while being computationally efficient and
adaptive to the rendering conditions during execution. Our model combines two
core ideas: (1) a fully convolutional architecture for decoding spatially
varying features, and (2) a rendering-adaptive per-pixel decoder. Both
techniques are integrated via a dense surface representation that is learned in
a weakly-supervised manner from low-topology mesh tracking over training
images. We demonstrate that PiCA improves reconstruction over existing
techniques across testing expressions and views on persons of different gender
and skin tone. Importantly, we show that the PiCA model is much smaller than
the state-of-art baseline model, and makes multi-person telecommunicaiton
possible: on a single Oculus Quest 2 mobile VR headset, 5 avatars are rendered
in realtime in the same scene.

本篇研究采用像素编码阿凡达（PiCA）作为一种深度生成模型，结合了卷积神经网络和自适应渲染的方法以实现在虚拟或增强现实中与逼真的人物形象进行面对面交流。实验表明，PiCA 比现有技术在不同性别和肤色的人物上的表情和视图的再现具有更好的表现，并且比当前基准模型更为紧凑，使得在单个移动 VR 头显上同时呈现 5 个人物形象成为可能。