We present Panoptic Neural Fields (PNF), an object-aware neural scene
representation that decomposes a scene into a set of objects (things) and
background (stuff). Each object is represented by an oriented 3D bounding box
and a multi-layer perceptron (MLP) that takes position, direction, and time and
outputs density and radiance. The background stuff is represented by a similar
MLP that additionally outputs semantic labels. Each object MLPs are
instance-specific and thus can be smaller and faster than previous object-aware
approaches, while still leveraging category-specific priors incorporated via
meta-learned initialization. Our model builds a panoptic radiance field
representation of any scene from just color images. We use off-the-shelf
algorithms to predict camera poses, object tracks, and 2D image semantic
segmentations. Then we jointly optimize the MLP weights and bounding box
parameters using analysis-by-synthesis with self-supervision from color images
and pseudo-supervision from predicted semantic segmentations. During
experiments with real-world dynamic scenes, we find that our model can be used
effectively for several tasks like novel view synthesis, 2D panoptic
segmentation, 3D scene editing, and multiview depth prediction.

我们提出了全视场神经场（PNF），这是一种对象感知的场景神经表示，可将场景分解为一组对象和背景。每个对象都由一个定向的 3D 边界框和一个多层感知器（MLP）表示，可以采取位置、方向和时间并输出密度和辐射率。