Rendering dynamic 3D human from monocular videos is crucial for various
applications such as virtual reality and digital entertainment. Most methods
assume the people is in an unobstructed scene, while various objects may cause
the occlusion of body parts in real-life scenarios. Previous method utilizing
NeRF for surface rendering to recover the occluded areas, but it requiring more
than one day to train and several seconds to render, failing to meet the
requirements of real-time interactive applications. To address these issues, we
propose OccGaussian based on 3D Gaussian Splatting, which can be trained within
6 minutes and produces high-quality human renderings up to 160 FPS with
occluded input. OccGaussian initializes 3D Gaussian distributions in the
canonical space, and we perform occlusion feature query at occluded regions,
the aggregated pixel-align feature is extracted to compensate for the missing
information. Then we use Gaussian Feature MLP to further process the feature
along with the occlusion-aware loss functions to better perceive the occluded
area. Extensive experiments both in simulated and real-world occlusions,
demonstrate that our method achieves comparable or even superior performance
compared to the state-of-the-art method. And we improving training and
inference speeds by 250x and 800x, respectively. Our code will be available for
research purposes.

我们提出了基于 3D 高斯散射的 OccGaussian 方法，其可在 6 分钟内进行训练，并且以高质量 160FPS 渲染具有遮挡输入的人体场景。通过在标准空间中初始化 3D 高斯分布，并在遮挡区域进行遮挡特征查询，提取聚合的像素对齐特征来弥补丢失的信息，然后我们使用高斯特征 MLP 进一步处理特征，并结合面向遮挡区域的损失函数来更好地感知遮挡区域。大量的实验结果表明，我们的方法在模拟和真实世界遮挡场景中实现了与最先进方法相媲美甚至更优的性能，且训练和推理速度提高了 250 倍和 800 倍，我们的代码将提供给研究目的使用。

OccGaussian：用于遮挡人体渲染的三维高斯粒子

OccGaussian: 3D Gaussian Splatting for Occluded Human Rendering

Facial expression analysis is an important tool for human-computer
interaction. In this paper, we introduce LibreFace, an open-source toolkit for
facial expression analysis. This open-source toolbox offers real-time and
offline analysis of facial behavior through deep learning models, including
facial action unit (AU) detection, AU intensity estimation, and facial
expression recognition. To accomplish this, we employ several techniques,
including the utilization of a large-scale pre-trained network, feature-wise
knowledge distillation, and task-specific fine-tuning. These approaches are
designed to effectively and accurately analyze facial expressions by leveraging
visual information, thereby facilitating the implementation of real-time
interactive applications. In terms of Action Unit (AU) intensity estimation, we
achieve a Pearson Correlation Coefficient (PCC) of 0.63 on DISFA, which is 7%
higher than the performance of OpenFace 2.0 while maintaining highly-efficient
inference that runs two times faster than OpenFace 2.0. Despite being compact,
our model also demonstrates competitive performance to state-of-the-art facial
expression analysis methods on AffecNet, FFHQ, and RAFDB. Our code will be
released at this https URL

通过使用深度学习模型和开源工具 LibreFace，我们能够实现面部表情分析，从而有效和准确地分析面部表情，并实现实时交互应用。