We propose RainyScape, an unsupervised framework for reconstructing clean
scenes from a collection of multi-view rainy images. RainyScape consists of two
main modules: a neural rendering module and a rain-prediction module that
incorporates a predictor network and a learnable latent embedding that captures
the rain characteristics of the scene. Specifically, based on the spectral bias
property of neural networks, we first optimize the neural rendering pipeline to
obtain a low-frequency scene representation. Subsequently, we jointly optimize
the two modules, driven by the proposed adaptive direction-sensitive
gradient-based reconstruction loss, which encourages the network to distinguish
between scene details and rain streaks, facilitating the propagation of
gradients to the relevant components. Extensive experiments on both the classic
neural radiance field and the recently proposed 3D Gaussian splatting
demonstrate the superiority of our method in effectively eliminating rain
streaks and rendering clean images, achieving state-of-the-art performance. The
constructed high-quality dataset and source code will be publicly available.

RainyScape 是一个无监督的框架，能够从多角度的雨天图像集合中重建干净的场景。该框架包含两个主要模块：神经渲染模块和雨预测模块，通过学习可捕捉场景雨水特征的预测网络和可学习的潜在嵌入，在优化神经渲染管道以获得低频场景表示后，使用自适应的方向敏感梯度重建损失来联合优化这两个模块，从而消除雨水痕迹并生成干净的图像，实验表明该方法在消除雨水痕迹和渲染干净图像方面具有卓越表现，达到了最先进的水平。