We propose GazeNeRF, a 3D-aware method for the task of gaze redirection. Existing gaze redirection methods operate on 2D images and struggle to generate 3D consistent results. Instead, we build on the intuition that the face region and eyeballs are separate 3D structures that move in a coordinated yet independent fashion. Our method leverages recent advancements in conditional image-based neural radiance fields and proposes a two-stream architecture that predicts volumetric features for the face and eye regions separately. Rigidly transforming the eye features via a 3D rotation matrix provides fine-grained control over the desired gaze angle. The final, redirected image is then attained via differentiable volume compositing. Our experiments show that this architecture outperforms naively conditioned NeRF baselines as well as previous state-of-the-art 2D gaze redirection methods in terms of redirection accuracy and identity preservation.

提出了GazeNeRF，一种针对凝视重定向任务的三维感知方法，使用条件图像神经辐射场预测面部和眼部区域的体特征，通过三维旋转矩阵刚性变换眼部特征，再经过可微体积合成得到最终的凝视重定向图像。实验表明，该方法在重定向精度和身份保留方面优于基线方法和以前最先进的2D凝视重定向方法。

GazeNeRF：使用神经辐射场进行三维感知的凝视重定向