Neural Radiance Field (NeRF) has shown impressive results in novel view
synthesis, particularly in Virtual Reality (VR) and Augmented Reality (AR),
thanks to its ability to represent scenes continuously. However, when just a
few input view images are available, NeRF tends to overfit the given views and
thus make the estimated depths of pixels share almost the same value. Unlike
previous methods that conduct regularization by introducing complex priors or
additional supervisions, we propose a simple yet effective method that
explicitly builds depth-aware consistency across input views to tackle this
challenge. Our key insight is that by forcing the same spatial points to be
sampled repeatedly in different input views, we are able to strengthen the
interactions between views and therefore alleviate the overfitting problem. To
achieve this, we build the neural networks on layered representations
(\textit{i.e.}, multiplane images), and the sampling point can thus be
resampled on multiple discrete planes. Furthermore, to regularize the unseen
target views, we constrain the rendered colors and depths from different input
views to be the same. Although simple, extensive experiments demonstrate that
our proposed method can achieve better synthesis quality over state-of-the-art
methods.

通过在 Neural Radiance Field 中引入深度感知一致性，利用分层表示构建多平面图像上的神经网络，并通过约束不同输入视图的渲染颜色和深度来规范未见目标视图，我们提出的方法能够在合成质量上超越现有方法。

CMC: 跨视层多平面一致性的少样本新视角合成

CMC: Few-shot Novel View Synthesis via Cross-view Multiplane Consistency

Novel view synthesis (NVS) of multi-human scenes imposes challenges due to
the complex inter-human occlusions. Layered representations handle the
complexities by dividing the scene into multi-layered radiance fields, however,
they are mainly constrained to per-scene optimization making them inefficient.
Generalizable human view synthesis methods combine the pre-fitted 3D human
meshes with image features to reach generalization, yet they are mainly
designed to operate on single-human scenes. Another drawback is the reliance on
multi-step optimization techniques for parametric pre-fitting of the 3D body
models that suffer from misalignment with the images in sparse view settings
causing hallucinations in synthesized views. In this work, we propose,
GenLayNeRF, a generalizable layered scene representation for free-viewpoint
rendering of multiple human subjects which requires no per-scene optimization
and very sparse views as input. We divide the scene into multi-human layers
anchored by the 3D body meshes. We then ensure pixel-level alignment of the
body models with the input views through a novel end-to-end trainable module
that carries out iterative parametric correction coupled with multi-view
feature fusion to produce aligned 3D models. For NVS, we extract point-wise
image-aligned and human-anchored features which are correlated and fused using
self-attention and cross-attention modules. We augment low-level RGB values
into the features with an attention-based RGB fusion module. To evaluate our
approach, we construct two multi-human view synthesis datasets; DeepMultiSyn
and ZJU-MultiHuman. The results indicate that our proposed approach outperforms
generalizable and non-human per-scene NeRF methods while performing at par with
layered per-scene methods without test time optimization.

提出了一种适用于多人场景的通用分层场景表示方法，用于多人视点渲染，无需每场景优化且只需要很少的输入视图。通过将场景分割为多人分层光辐射场，并通过迭代参数修正和多视图特征融合实现与输入视图的像素级对齐的 3D 模型，进一步提出了新颖的端到端训练模块进行人体模型校正。在新颖视图综合和非人物优化场景 NeRF 方法上表现优越，而与分层场景方法在测试时间优化方面相当。

GenLayNeRF: 用于多人视角合成的可泛化分层表示与 3D 模型对齐

GenLayNeRF: Generalizable Layered Representations with 3D Model  Alignment for Multi-Human View Synthesis

Diffusion-based methods can generate realistic images and videos, but they
struggle to edit existing objects in a video while preserving their appearance
over time. This prevents diffusion models from being applied to natural video
editing in practical scenarios. In this paper, we tackle this problem by
introducing temporal dependency to existing text-driven diffusion models, which
allows them to generate consistent appearance for the edited objects.
Specifically, we develop a novel inter-frame propagation mechanism for
diffusion video editing, which leverages the concept of layered representations
to propagate the appearance information from one frame to the next. We then
build up a text-driven video editing framework based on this mechanism, namely
StableVideo, which can achieve consistency-aware video editing. Extensive
experiments demonstrate the strong editing capability of our approach. Compared
with state-of-the-art video editing methods, our approach shows superior
qualitative and quantitative results. Our code is available at
\href{this https URL}{this https URL}.

通过引入时态依赖于现有的文本驱动扩散模型，使其能够生成一致的编辑对象外观，我们解决了扩散模型在自然视频编辑中编辑现有对象时难以保持其外观随时间稳定的问题。通过开发一种新颖的帧间传播机制，利用分层表示的概念将相邻帧的外观信息传播到下一帧，并基于该机制构建了一个名为 StableVideo 的文本驱动视频编辑框架，可以实现一致感知的视频编辑。广泛的实验证明了我们方法的强大编辑能力。与最先进的视频编辑方法相比，我们方法展示了卓越的定性和定量结果。