Image translation and manipulation have gain increasing attention along with
the rapid development of deep generative models. Although existing approaches
have brought impressive results, they mainly operated in 2D space. In light of
recent advances in NeRF-based 3D-aware generative models, we introduce a new
task, Semantic-to-NeRF translation, that aims to reconstruct a 3D scene
modelled by NeRF, conditioned on one single-view semantic mask as input. To
kick-off this novel task, we propose the Sem2NeRF framework. In particular,
Sem2NeRF addresses the highly challenging task by encoding the semantic mask
into the latent code that controls the 3D scene representation of a pre-trained
decoder. To further improve the accuracy of the mapping, we integrate a new
region-aware learning strategy into the design of both the encoder and the
decoder. We verify the efficacy of the proposed Sem2NeRF and demonstrate that
it outperforms several strong baselines on two benchmark datasets. Code and
video are available at this https URL

本文提出了一个名为 Sem2NeRF 的框架，实现了从单视角语义掩码到 3D 物体模型的转换任务，并采用区域感知学习策略对映射的精度进行改进，其在两个基准数据集上表现优于多个强基线模型。