Point-based representations have recently gained popularity in novel view
synthesis, for their unique advantages, e.g., intuitive geometric
representation, simple manipulation, and faster convergence. However, based on
our observation, these point-based neural re-rendering methods are only
expected to perform well under ideal conditions and suffer from noisy, patchy
points and unbounded scenes, which are challenging to handle but defacto common
in real applications. To this end, we revisit one such influential method,
known as Neural Point-based Graphics (NPBG), as our baseline, and propose
Robust Point-based Graphics (RPBG). We in-depth analyze the factors that
prevent NPBG from achieving satisfactory renderings on generic datasets, and
accordingly reform the pipeline to make it more robust to varying datasets
in-the-wild. Inspired by the practices in image restoration, we greatly enhance
the neural renderer to enable the attention-based correction of point
visibility and the inpainting of incomplete rasterization, with only acceptable
overheads. We also seek for a simple and lightweight alternative for
environment modeling and an iterative method to alleviate the problem of poor
geometry. By thorough evaluation on a wide range of datasets with different
shooting conditions and camera trajectories, RPBG stably outperforms the
baseline by a large margin, and exhibits its great robustness over
state-of-the-art NeRF-based variants. Code available at
this https URL

经深入分析 Neural Point-based Graphics (NPBG) 方法在通用数据集上无法实现令人满意的渲染效果，本研究提出了 Robust Point-based Graphics (RPBG) 方法，通过启发图像修复方法，大大增强了神经渲染器的能力，使其能够根据点的可见性进行修正，并补全不完整的光栅化，同时，通过简单轻量的环境建模和迭代方法，缓解了几何问题。经过对不同拍摄条件和摄像机轨迹的广泛评估，RPBG 在性能上明显优于基线方法，并展现了其对最新的 NeRF-based 变体的强大鲁棒性。

RPBG：面向野外环境的鲁棒神经点图形

RPBG: Towards Robust Neural Point-based Graphics in the Wild

Real world images often have highly imbalanced content density. Some areas
are very uniform, e.g., large patches of blue sky, while other areas are
scattered with many small objects. Yet, the commonly used successive grid
downsampling strategy in convolutional deep networks treats all areas equally.
Hence, small objects are represented in very few spatial locations, leading to
worse results in tasks such as segmentation. Intuitively, retaining more pixels
representing small objects during downsampling helps to preserve important
information. To achieve this, we propose AutoFocusFormer (AFF), a
local-attention transformer image recognition backbone, which performs adaptive
downsampling by learning to retain the most important pixels for the task.
Since adaptive downsampling generates a set of pixels irregularly distributed
on the image plane, we abandon the classic grid structure. Instead, we develop
a novel point-based local attention block, facilitated by a balanced clustering
module and a learnable neighborhood merging module, which yields
representations for our point-based versions of state-of-the-art segmentation
heads. Experiments show that our AutoFocusFormer (AFF) improves significantly
over baseline models of similar sizes.

本研究提出了一种名为 AutoFocusFormer (AFF) 的局部 - 注意力 Transformer 图像识别骨干网络，通过学习保留任务中最重要的像素来实现自适应降采样，采用点基础分割方法，使得小目标在降采样过程中得到更充分地表示，并展示它在分割任务中显著优于基线模型。