3D scene stylization aims at generating stylized images of the scene from
arbitrary novel views following a given set of style examples, while ensuring
consistency when rendered from different views. Directly applying methods for
image or video stylization to 3D scenes cannot achieve such consistency. Thanks
to recently proposed neural radiance fields (NeRF), we are able to represent a
3D scene in a consistent way. Consistent 3D scene stylization can be
effectively achieved by stylizing the corresponding NeRF. However, there is a
significant domain gap between style examples which are 2D images and NeRF
which is an implicit volumetric representation. To address this problem, we
propose a novel mutual learning framework for 3D scene stylization that
combines a 2D image stylization network and NeRF to fuse the stylization
ability of 2D stylization network with the 3D consistency of NeRF. We first
pre-train a standard NeRF of the 3D scene to be stylized and replace its color
prediction module with a style network to obtain a stylized NeRF. It is
followed by distilling the prior knowledge of spatial consistency from NeRF to
the 2D stylization network through an introduced consistency loss. We also
introduce a mimic loss to supervise the mutual learning of the NeRF style
module and fine-tune the 2D stylization decoder. In order to further make our
model handle ambiguities of 2D stylization results, we introduce learnable
latent codes that obey the probability distributions conditioned on the style.
They are attached to training samples as conditional inputs to better learn the
style module in our novel stylized NeRF. Experimental results demonstrate that
our method is superior to existing approaches in both visual quality and
long-range consistency.

本文提出了一种新的双向学习框架用于 3D 场景风格化，结合 2D 图像风格化网络和 NeRF，并引入了一种一致性损失函数来迁移 NeRF 的一致性先验知识到 2D 风格化网络中。实验结果表明，这种方法在视觉质量和长程一致性方面优于现有方法。