Dynamic neural radiance fields (dynamic NeRFs) have demonstrated impressive
results in novel view synthesis on 3D dynamic scenes. However, they often
require complete video sequences for training followed by novel view synthesis,
which is similar to playing back the recording of a dynamic 3D scene. In
contrast, we propose OD-NeRF to efficiently train and render dynamic NeRFs
on-the-fly which instead is capable of streaming the dynamic scene. When
training on-the-fly, the training frames become available sequentially and the
model is trained and rendered frame-by-frame. The key challenge of efficient
on-the-fly training is how to utilize the radiance field estimated from the
previous frames effectively. To tackle this challenge, we propose: 1) a NeRF
model conditioned on the multi-view projected colors to implicitly track
correspondence between the current and previous frames, and 2) a transition and
update algorithm that leverages the occupancy grid from the last frame to
sample efficiently at the current frame. Our algorithm can achieve an
interactive speed of 6FPS training and rendering on synthetic dynamic scenes
on-the-fly, and a significant speed-up compared to the state-of-the-art on
real-world dynamic scenes.

提出了 OD-NeRF，它可以实现动态三维场景的流式训练和渲染，并在实现帧间隐式跟踪和采样方面取得了重要进展。

OD-NeRF：实时动态神经辐射场的高效训练

OD-NeRF: Efficient Training of On-the-Fly Dynamic Neural Radiance Fields

Replay in neural networks involves training on sequential data with memorized
samples, which counteracts forgetting of previous behavior caused by
non-stationarity. We present a method where these auxiliary samples are
generated on the fly, given only the model that is being trained for the
assessed objective, without extraneous buffers or generator networks. Instead
the implicit memory of learned samples within the assessed model itself is
exploited. Furthermore, whereas existing work focuses on reinforcing the full
seen data distribution, we show that optimizing for not forgetting calls for
the generation of samples that are specialized to each real training batch,
which is more efficient and scalable. We consider high-level parallels with the
brain, notably the use of a single model for inference and recall, the
dependency of recalled samples on the current environment batch, top-down
modulation of activations and learning, abstract recall, and the dependency
between the degree to which a task is learned and the degree to which it is
recalled. These characteristics emerge naturally from the method without being
controlled for.

本篇文章提出了一种新的方法来缓解训练神经网络时非稳态带来的行为遗忘现象，利用模型内部的内隐记忆来进行实时样本生成，并侧重于优化实际的训练数据批次，从而高效和可扩展的进行训练。作者探讨了该方法与大脑模型的对应之处，并发现这种方法的高效特性是自然而然地产生的。