Reconstructing and editing 3D objects and scenes both play crucial roles in
computer graphics and computer vision. Neural radiance fields (NeRFs) can
achieve realistic reconstruction and editing results but suffer from
inefficiency in rendering. Gaussian splatting significantly accelerates
rendering by rasterizing Gaussian ellipsoids. However, Gaussian splatting
utilizes a single Spherical Harmonic (SH) function to model both texture and
lighting, limiting independent editing capabilities of these components.
Recently, attempts have been made to decouple texture and lighting with the
Gaussian splatting representation but may fail to produce plausible geometry
and decomposition results on reflective scenes. Additionally, the forward
shading technique they employ introduces noticeable blending artifacts during
relighting, as the geometry attributes of Gaussians are optimized under the
original illumination and may not be suitable for novel lighting conditions. To
address these issues, we introduce DeferredGS, a method for decoupling and
editing the Gaussian splatting representation using deferred shading. To
achieve successful decoupling, we model the illumination with a learnable
environment map and define additional attributes such as texture parameters and
normal direction on Gaussians, where the normal is distilled from a jointly
trained signed distance function. More importantly, we apply deferred shading,
resulting in more realistic relighting effects compared to previous methods.
Both qualitative and quantitative experiments demonstrate the superior
performance of DeferredGS in novel view synthesis and editing tasks.

通过 DeferredGS 方法，采用延迟着色技术，模拟环境光照并定义额外的属性对高斯粒子进行解耦和编辑，从而实现更逼真的重新光照效果，并在新视角合成和编辑任务方面表现出卓越性能。

DeferredGS：具有延迟渲染和可编辑高斯喷点的解耦设计

DeferredGS: Decoupled and Editable Gaussian Splatting with Deferred  Shading

Visual error metrics play a fundamental role in the quantification of
perceived image similarity. Most recently, use cases for them in real-time
applications have emerged, such as content-adaptive shading and shading reuse
to increase performance and improve efficiency. A wide range of different
metrics has been established, with the most sophisticated being capable of
capturing the perceptual characteristics of the human visual system. However,
their complexity, computational expense, and reliance on reference images to
compare against prevent their generalized use in real-time, restricting such
applications to using only the simplest available metrics. In this work, we
explore the abilities of convolutional neural networks to predict a variety of
visual metrics without requiring either reference or rendered images.
Specifically, we train and deploy a neural network to estimate the visual error
resulting from reusing shading or using reduced shading rates. The resulting
models account for 70%-90% of the variance while achieving up to an order of
magnitude faster computation times. Our solution combines image-space
information that is readily available in most state-of-the-art deferred shading
pipelines with reprojection from previous frames to enable an adequate estimate
of visual errors, even in previously unseen regions. We describe a suitable
convolutional network architecture and considerations for data preparation for
training. We demonstrate the capability of our network to predict complex error
metrics at interactive rates in a real-time application that implements
content-adaptive shading in a deferred pipeline. Depending on the portion of
unseen image regions, our approach can achieve up to $2\times$ performance
compared to state-of-the-art methods.

利用卷积神经网络预测视觉错误指标使得实时应用中的内容自适应阴影渲染更高效，具备较强的可推广性和计算速度。