We present a novel differentiable point-based rendering framework for
material and lighting decomposition from multi-view images, enabling editing,
ray-tracing, and real-time relighting of the 3D point cloud. Specifically, a 3D
scene is represented as a set of relightable 3D Gaussian points, where each
point is additionally associated with a normal direction, BRDF parameters, and
incident lights from different directions. To achieve robust lighting
estimation, we further divide incident lights of each point into global and
local components, as well as view-dependent visibilities. The 3D scene is
optimized through the 3D Gaussian Splatting technique while BRDF and lighting
are decomposed by physically-based differentiable rendering. Moreover, we
introduce an innovative point-based ray-tracing approach based on the bounding
volume hierarchy for efficient visibility baking, enabling real-time rendering
and relighting of 3D Gaussian points with accurate shadow effects. Extensive
experiments demonstrate improved BRDF estimation and novel view rendering
results compared to state-of-the-art material estimation approaches. Our
framework showcases the potential to revolutionize the mesh-based graphics
pipeline with a relightable, traceable, and editable rendering pipeline solely
based on point cloud. Project
page:this https URL

我们提出了一种新颖的可微分的基于点的渲染框架，用于从多视图图像中进行材料和光照分解，实现对 3D 点云的编辑、光线跟踪和实时换光。