To represent people in mixed reality applications for collaboration and
communication, we need to generate realistic and faithful avatar poses.
However, the signal streams that can be applied for this task from head-mounted
devices (HMDs) are typically limited to head pose and hand pose estimates.
While these signals are valuable, they are an incomplete representation of the
human body, making it challenging to generate a faithful full-body avatar. We
address this challenge by developing a flow-based generative model of the 3D
human body from sparse observations, wherein we learn not only a conditional
distribution of 3D human pose, but also a probabilistic mapping from
observations to the latent space from which we can generate a plausible pose
along with uncertainty estimates for the joints. We show that our approach is
not only a strong predictive model, but can also act as an efficient pose prior
in different optimization settings where a good initial latent code plays a
major role.

针对在协作和交流中需要产生逼真可信的虚拟人物角色，本文提出了一种基于流的 3D 人体生成模型，通过学习 3D 人体姿势的条件分布和从观察到的数据到潜在空间的概率映射，生成能够确定关节不确定度的虚拟角色动作，证明了该方法不仅是一个强大的预测模型，还可以在各种优化环境中充当一个高效的姿势先验。

基于流的稀疏观测生成三维头像

FLAG: Flow-based 3D Avatar Generation from Sparse Observations

Hand gestures form an intuitive means of interaction in Mixed Reality (MR)
applications. However, accurate gesture recognition can be achieved only
through state-of-the-art deep learning models or with the use of expensive
sensors. Despite the robustness of these deep learning models, they are
generally computationally expensive and obtaining real-time performance
on-device is still a challenge. To this end, we propose a novel lightweight
hand gesture recognition framework that works in First Person View for wearable
devices. The models are trained on a GPU machine and ported on an Android
smartphone for its use with frugal wearable devices such as the Google
Cardboard and VR Box. The proposed hand gesture recognition framework is driven
by a cascade of state-of-the-art deep learning models: MobileNetV2 for hand
localisation, our custom fingertip regression architecture followed by a
Bi-LSTM model for gesture classification. We extensively evaluate the framework
on our EgoGestAR dataset. The overall framework works in real-time on mobile
devices and achieves a classification accuracy of 80% on EgoGestAR video
dataset with an average latency of only 0.12 s.

我们提出了一种用于便携式可穿戴设备的新型轻量级手势识别框架，该框架由一系列最先进的深度学习模型驱动，能以实时性灵敏地识别手势，实现了 80％的分类准确度和仅 0.12 秒的平均延迟。

GestARLite：一种基于指向手指的智能手机和视频透视耳机的设备内手势界面

GestARLite: An On-Device Pointing Finger Based Gestural Interface for  Smartphones and Video See-Through Head-Mounts

We present the first end to end approach for real time material estimation
for general object shapes with uniform material that only requires a single
color image as input. In addition to Lambertian surface properties, our
approach fully automatically computes the specular albedo, material shininess,
and a foreground segmentation. We tackle this challenging and ill posed inverse
rendering problem using recent advances in image to image translation
techniques based on deep convolutional encoder decoder architectures. The
underlying core representations of our approach are specular shading, diffuse
shading and mirror images, which allow to learn the effective and accurate
separation of diffuse and specular albedo. In addition, we propose a novel
highly efficient perceptual rendering loss that mimics real world image
formation and obtains intermediate results even during run time. The estimation
of material parameters at real time frame rates enables exciting mixed reality
applications, such as seamless illumination consistent integration of virtual
objects into real world scenes, and virtual material cloning. We demonstrate
our approach in a live setup, compare it to the state of the art, and
demonstrate its effectiveness through quantitative and qualitative evaluation.

该研究提出了实时材料估计的端到端方法，使用深度卷积编码器解码器结构的图像到图像翻译技术，精确获取漫反射和镜面高光的分离，并使用新的高效感知渲染损失，使实时视频中的虚拟物体拟真融合。