3D Gaussian Splatting-based techniques have recently advanced 3D scene
reconstruction and novel view synthesis, achieving high-quality real-time
rendering. However, these approaches are inherently limited by the underlying
pinhole camera assumption in modeling the images and hence only work for
All-in-Focus (AiF) sharp image inputs. This severely affects their
applicability in real-world scenarios where images often exhibit defocus blur
due to the limited depth-of-field (DOF) of imaging devices. Additionally,
existing 3D Gaussian Splatting (3DGS) methods also do not support rendering of
DOF effects.
To address these challenges, we introduce DOF-GS that allows for rendering
adjustable DOF effects, removing defocus blur as well as refocusing of 3D
scenes, all from multi-view images degraded by defocus blur. To this end, we
re-imagine the traditional Gaussian Splatting pipeline by employing a finite
aperture camera model coupled with explicit, differentiable defocus rendering
guided by the Circle-of-Confusion (CoC). The proposed framework provides for
dynamic adjustment of DOF effects by changing the aperture and focal distance
of the underlying camera model on-demand. It also enables rendering varying DOF
effects of 3D scenes post-optimization, and generating AiF images from
defocused training images. Furthermore, we devise a joint optimization strategy
to further enhance details in the reconstructed scenes by jointly optimizing
rendered defocused and AiF images. Our experimental results indicate that
DOF-GS produces high-quality sharp all-in-focus renderings conditioned on
inputs compromised by defocus blur, with the training process incurring only a
modest increase in GPU memory consumption. We further demonstrate the
applications of the proposed method for adjustable defocus rendering and
refocusing of the 3D scene from input images degraded by defocus blur.

DOF-GS 是一种基于 3D 高斯喷洒技术的方法，可以从被虚化的多视图图像中实现可调焦模糊效果和三维场景重新聚焦。

DOF-GS：可调焦深度 3D 高斯喷墨在重新聚焦、散焦渲染和去模糊中的应用

DOF-GS: Adjustable Depth-of-Field 3D Gaussian Splatting for  Refocusing,Defocus Rendering and Blur Removal

Neural Radiance Field (NeRF) and its variants have exhibited great success on
representing 3D scenes and synthesizing photo-realistic novel views. However,
they are generally based on the pinhole camera model and assume all-in-focus
inputs. This limits their applicability as images captured from the real world
often have finite depth-of-field (DoF). To mitigate this issue, we introduce
DoF-NeRF, a novel neural rendering approach that can deal with shallow DoF
inputs and can simulate DoF effect. In particular, it extends NeRF to simulate
the aperture of lens following the principles of geometric optics. Such a
physical guarantee allows DoF-NeRF to operate views with different focus
configurations. Benefiting from explicit aperture modeling, DoF-NeRF also
enables direct manipulation of DoF effect by adjusting virtual aperture and
focus parameters. It is plug-and-play and can be inserted into NeRF-based
frameworks. Experiments on synthetic and real-world datasets show that,
DoF-NeRF not only performs comparably with NeRF in the all-in-focus setting,
but also can synthesize all-in-focus novel views conditioned on shallow DoF
inputs. An interesting application of DoF-NeRF to DoF rendering is also
demonstrated. The source code will be made available at
this https URL

介绍了一种称为 DoF-NeRF 的新颖神经渲染方法，能够处理浅景深图像输入并模拟景深效果，通过直接调整虚拟光圈和对焦参数可以进行景深效果的直接操作。

DoF-NeRF: 深度场景与神经辐射场相遇

DoF-NeRF: Depth-of-Field Meets Neural Radiance Fields

We present a controllable camera simulator based on deep neural networks to
synthesize raw image data under different camera settings, including exposure
time, ISO, and aperture. The proposed simulator includes an exposure module
that utilizes the principle of modern lens designs for correcting the luminance
level. It also contains a noise module using the noise level function and an
aperture module with adaptive attention to simulate the side effects on noise
and defocus blur. To facilitate the learning of a simulator model, we collect a
dataset of the 10,000 raw images of 450 scenes with different exposure
settings. Quantitative experiments and qualitative comparisons show that our
approach outperforms relevant baselines in raw data synthesize on multiple
cameras. Furthermore, the camera simulator enables various applications,
including large-aperture enhancement, HDR, auto exposure, and data augmentation
for training local feature detectors. Our work represents the first attempt to
simulate a camera sensor's behavior leveraging both the advantage of
traditional raw sensor features and the power of data-driven deep learning.

本论文提出了一个基于深度神经网络的可控相机模拟器，能够在不同的相机设置下合成原始图像数据，包括曝光时间、ISO、光圈等，并且能够用于大光圈增强、HDR、自动曝光和数据增强等多个应用领域。同时，也是第一个利用传统原始传感器特征和数据驱动深度学习优势模拟相机传感器行为的尝试。