Advances in NERFs have allowed for 3D scene reconstructions and novel view
synthesis. Yet, efficiently editing these representations while retaining
photorealism is an emerging challenge. Recent methods face three primary
limitations: they're slow for interactive use, lack precision at object
boundaries, and struggle to ensure multi-view consistency. We introduce IReNe
to address these limitations, enabling swift, near real-time color editing in
NeRF. Leveraging a pre-trained NeRF model and a single training image with
user-applied color edits, IReNe swiftly adjusts network parameters in seconds.
This adjustment allows the model to generate new scene views, accurately
representing the color changes from the training image while also controlling
object boundaries and view-specific effects. Object boundary control is
achieved by integrating a trainable segmentation module into the model. The
process gains efficiency by retraining only the weights of the last network
layer. We observed that neurons in this layer can be classified into those
responsible for view-dependent appearance and those contributing to diffuse
appearance. We introduce an automated classification approach to identify these
neuron types and exclusively fine-tune the weights of the diffuse neurons. This
further accelerates training and ensures consistent color edits across
different views. A thorough validation on a new dataset, with edited object
colors, shows significant quantitative and qualitative advancements over
competitors, accelerating speeds by 5x to 500x.

通过在预先训练的 NERF 模型和单一训练图像中实时调整网络参数，IReNe 能够快速、准确地调整颜色编辑，同时控制物体边界和视图效果，从而实现了在 NERF 中近实时的颜色编辑和物体控制。

IReNe: 神经辐射场中的即时染色

IReNe: Instant Recoloring in Neural Radiance Fields

We develop a new method for portrait image editing, which supports
fine-grained editing of geometries, colors, lights and shadows using a single
neural network model. We adopt a novel asymmetric conditional GAN architecture:
the generators take the transformed conditional inputs, such as edge maps,
color palette, sliders and masks, that can be directly edited by the user; the
discriminators take the conditional inputs in the way that can guide
controllable image generation more effectively. Taking color editing as an
example, we feed color palettes (which can be edited easily) into the
generator, and color maps (which contain positional information of colors) into
the discriminator. We also design a region-weighted discriminator so that
higher weights are assigned to more important regions, like eyes and skin.
Using a color palette, the user can directly specify the desired colors of
hair, skin, eyes, lip and background. Color sliders allow the user to blend
colors in an intuitive manner. The user can also edit lights and shadows by
modifying the corresponding masks. We demonstrate the effectiveness of our
method by evaluating it on the CelebAMask-HQ dataset with a wide range of
tasks, including geometry/color/shadow/light editing, hand-drawn sketch to
image translation, and color transfer. We also present ablation studies to
justify our design.

本文提出了一种新的人像图像编辑方法，采用不对称条件 GAN 架构，支持几何、颜色、光和阴影的细粒度编辑。通过向生成器提供可以直接由用户编辑的条件输入来实现编辑，例如边缘图、颜色板、滑块和蒙版。该方法具有较好的编辑效果，并且支持几何、颜色、阴影、光的多种编辑方式，同时也支持从手绘草图到图像的转换和颜色转换等任务。