DensePose provides a pixel-accurate association of images with 3D mesh
coordinates, but does not provide a 3D mesh, while Human Mesh Reconstruction
(HMR) systems have high 2D reprojection error, as measured by DensePose
localization metrics. In this work we introduce MeshPose to jointly tackle
DensePose and HMR. For this we first introduce new losses that allow us to use
weak DensePose supervision to accurately localize in 2D a subset of the mesh
vertices ('VertexPose'). We then lift these vertices to 3D, yielding a low-poly
body mesh ('MeshPose'). Our system is trained in an end-to-end manner and is
the first HMR method to attain competitive DensePose accuracy, while also being
lightweight and amenable to efficient inference, making it suitable for
real-time AR applications.

DensePose 提供了图像与 3D 网格坐标的像素精确关联，而 Human Mesh Reconstruction（HMR）系统在 DensePose 定位度量中具有较高的 2D 重新投影误差。本文介绍了 MeshPose 并共同解决了 DensePose 和 HMR 问题。通过引入新的损失函数，我们使用弱 DensePose 监督准确定位部分网格顶点（'VertexPose'），然后将这些顶点提升到 3D，得到低多边形身体网格（'MeshPose'）。我们的系统以端到端方式进行训练，是第一个在保持竞争力的 DensePose 准确性的同时，也是轻量级且适用于高效推断的 HMR 方法，因此非常适合实时增强现实应用。

MeshPose：将 DensePose 和三维身体网格重建统一

MeshPose: Unifying DensePose and 3D Body Mesh reconstruction

We present a pipeline for realistic embedding of virtual objects into footage
of indoor scenes with focus on real-time AR applications. Our pipeline consists
of two main components: A light estimator and a neural soft shadow texture
generator. Our light estimation is based on deep neural nets and determines the
main light direction, light color, ambient color and an opacity parameter for
the shadow texture. Our neural soft shadow method encodes object-based
realistic soft shadows as light direction dependent textures in a small MLP. We
show that our pipeline can be used to integrate objects into AR scenes in a new
level of realism in real-time. Our models are small enough to run on current
mobile devices. We achieve runtimes of 9ms for light estimation and 5ms for
neural shadows on an iPhone 11 Pro.

我们提出了一个用于将虚拟对象逼真地嵌入到室内场景中的流程，重点关注实时增强现实应用。我们的流程包括两个主要组件：光估计器和神经软阴影纹理生成器。我们的光估计基于深度神经网络，确定主光方向、光颜色、环境颜色和阴影纹理的不透明度参数。我们的神经软阴影方法将基于物体的逼真软阴影编码为与光方向相关的纹理，存储在一个小型的多层感知机中。我们展示了我们的流程可以在实时中将对象融入增强现实场景中，实现新的逼真程度。我们的模型足够小，可以在现有的移动设备上运行，我们在 iPhone 11 Pro 上实现了 9ms 的光估计和 5ms 的神经阴影运行时间。

AR 室内场景的实时光估计与神经软阴影

Real-time Light Estimation and Neural Soft Shadows for AR Indoor  Scenarios

We present an end-to-end neural network-based model for inferring an
approximate 3D mesh representation of a human face from single camera input for
AR applications. The relatively dense mesh model of 468 vertices is well-suited
for face-based AR effects. The proposed model demonstrates super-realtime
inference speed on mobile GPUs (100-1000+ FPS, depending on the device and
model variant) and a high prediction quality that is comparable to the variance
in manual annotations of the same image.

本文提出了一种基于神经网络的端到端模型，从单个摄像头输入中推断出人脸的近似 3D 网格表示，适用于 AR 应用程序。该模型具有相对密集的 468 个顶点网格模型，非常适合基于人脸的 AR 效果。该模型在移动 GPU 上具有超级实时推断速度（100-1000+FPS，取决于设备和模型变体），并且预测质量与同一图像的手动注释的变化相当。