We propose INFAMOUS-NeRF, an implicit morphable face model that introduces
hypernetworks to NeRF to improve the representation power in the presence of
many training subjects. At the same time, INFAMOUS-NeRF resolves the classic
hypernetwork tradeoff of representation power and editability by learning
semantically-aligned latent spaces despite the subject-specific models, all
without requiring a large pretrained model. INFAMOUS-NeRF further introduces a
novel constraint to improve NeRF rendering along the face boundary. Our
constraint can leverage photometric surface rendering and multi-view
supervision to guide surface color prediction and improve rendering near the
surface. Finally, we introduce a novel, loss-guided adaptive sampling method
for more effective NeRF training by reducing the sampling redundancy. We show
quantitatively and qualitatively that our method achieves higher representation
power than prior face modeling methods in both controlled and in-the-wild
settings. Code and models will be released upon publication.

我们提出了 INFAMOUS-NeRF，一种隐式可塑性人脸模型，通过引入超网络到 NeRF 中，改善在训练多个主题时的表示能力。同时，INFAMOUS-NeRF 通过学习语义对齐的潜在空间来解决了经典超网络在表示能力和可编辑性之间的权衡问题，而无需大规模预训练模型。INFAMOUS-NeRF 还引入了一种改进 NeRF 渲染的新约束，该约束可以利用光度表面渲染和多视角监督来指导表面颜色预测并改善表面附近的渲染。最后，我们引入了一种新的、通过减少采样冗余来实现更有效 NeRF 训练的损失引导自适应采样方法。我们定量和定性地展示了我们的方法在受控和野外环境中实现了比先前人脸建模方法更高的表示能力。代码和模型将在发表后发布。

INFAMOUS-NeRF: 利用语义对齐的超网络和神经辐射场改进面部建模

INFAMOUS-NeRF: ImproviNg FAce MOdeling Using Semantically-Aligned  Hypernetworks with Neural Radiance Fields

This work delves into the task of pose-free novel view synthesis from stereo
pairs, a challenging and pioneering task in 3D vision. Our innovative
framework, unlike any before, seamlessly integrates 2D correspondence matching,
camera pose estimation, and NeRF rendering, fostering a synergistic enhancement
of these tasks. We achieve this through designing an architecture that utilizes
a shared representation, which serves as a foundation for enhanced 3D geometry
understanding. Capitalizing on the inherent interplay between the tasks, our
unified framework is trained end-to-end with the proposed training strategy to
improve overall model accuracy. Through extensive evaluations across diverse
indoor and outdoor scenes from two real-world datasets, we demonstrate that our
approach achieves substantial improvement over previous methodologies,
especially in scenarios characterized by extreme viewpoint changes and the
absence of accurate camera poses.

这项工作涉及从立体对生成无姿态新视角合成的任务，在 3D 视觉中是一个具有挑战性和开创性的任务。通过设计一个利用共享表示的架构，无缝集成 2D 对应匹配、相机姿态估计和 NeRF 渲染，促进这些任务的协同增强，我们在整个模型准确性上通过提议的训练策略进行端到端的培训。通过对两个现实世界数据集的多样化室内和室外场景进行广泛评估，我们证明我们的方法在特征视角变化和缺乏精确相机姿态的场景中实现了相当大的改进。