We present a system that automatically brings street view imagery to life by
populating it with naturally behaving, animated pedestrians and vehicles. Our
approach is to remove existing people and vehicles from the input image, insert
moving objects with proper scale, angle, motion, and appearance, plan paths and
traffic behavior, as well as render the scene with plausible occlusion and
shadowing effects. The system achieves these by reconstructing the still image
street scene, simulating crowd behavior, and rendering with consistent
lighting, visibility, occlusions, and shadows. We demonstrate results on a
diverse range of street scenes including regular still images and panoramas.

我们提出了一个系统，通过在现有的街景图像上插入自然行为的动画行人和车辆，使其活灵活现。我们的方法是将输入图像中的现有人和车辆移除，插入具有适当比例、角度、运动和外观的移动对象，规划路径和交通行为，并以合理的遮挡和阴影效果渲染场景。该系统通过重建静止图像街景、模拟人群行为以及以一致的光照、可见性、遮挡和阴影进行渲染来实现这些目标。我们在包括常规静态图像和全景图在内的多种街景场景上展示了结果。

街景动画

Animating Street View

Most NeRF-based models are designed for learning the entire scene, and
complex scenes can lead to longer learning times and poorer rendering effects.
This paper utilizes scene semantic priors to make improvements in fast
training, allowing the network to focus on the specific targets and not be
affected by complex backgrounds. The training speed can be increased by 7.78
times with better rendering effect, and small to medium sized targets can be
rendered faster. In addition, this improvement applies to all NeRF-based
models. Considering the inherent multi-view consistency and smoothness of NeRF,
this paper also studies weak supervision by sparsely sampling negative ray
samples. With this method, training can be further accelerated and rendering
quality can be maintained. Finally, this paper extends pixel semantic and color
rendering formulas and proposes a new scene editing technique that can achieve
unique displays of the specific semantic targets or masking them in rendering.
To address the problem of unsupervised regions incorrect inferences in the
scene, we also designed a self-supervised loop that combines morphological
operations and clustering.

通过利用场景语义先验，加快训练速度和提高渲染效果，本文改进了 NeRF-based 模型，在特定目标上进行快速训练，并且对所有 NeRF-based 模型都适用。此外，通过稀疏采样负射线样本并研究弱监督方法，进一步加速训练并保持渲染质量。最后，通过扩展像素语义和颜色渲染公式，提出了一种新的场景编辑技术，以实现特定语义目标的独特显示或在渲染中进行掩盖。同时，通过结合形态学运算和聚类，设计了自监督循环来解决场景中无监督区域的不正确推断问题。