In this paper, we develop a personalized video relighting algorithm that
produces high-quality and temporally consistent relit video under any pose,
expression, and lighting conditions in real-time. Existing relighting
algorithms typically rely either on publicly available synthetic data, which
yields poor relighting results, or instead on Light Stage data which is
inaccessible and is not publicly available. We show that by casually capturing
video of a user watching YouTube videos on a monitor we can train a
personalized algorithm capable of producing high-quality relighting under any
condition. Our key contribution is a novel neural relighting architecture that
effectively separates the intrinsic appearance features, geometry and
reflectance, from the source lighting and then combines it with the target
lighting to generate a relit image. This neural architecture enables smoothing
of intrinsic appearance features leading to temporally stable video relighting.
Both qualitative and quantitative evaluations show that our relighting
architecture improves portrait image relighting quality and temporal
consistency over state-of-the-art approaches on both casually captured Light
Stage at Your Desk (LSYD) data and Light Stage captured One Light At a Time
(OLAT) datasets.

我们开发了一种个性化视频照明算法，在任何姿势、表情和照明条件下实时生成高质量和时间一致的照亮视频。通过随意捕捉用户在监视器上观看 YouTube 视频的视频，我们展示了可以训练一种个性化算法，能够在任何条件下产生高质量的照明。我们的关键贡献是一种新颖的神经照明架构，能够有效地将内在的外观特征、几何和反射从源照明中分离出来，并将其与目标照明结合起来生成一个照亮图像。这种神经架构能够平滑内在的外观特征，从而实现时间稳定的视频照明。定性和定量评估表明，我们的照明架构在随意捕捉的 Light Stage at Your Desk（LSYD）数据和一次只捕捉一个光的 Light Stage (OLAT) 数据集上，比最先进的方法改进了肖像图像的照明质量和时间一致性。