Video generation models have demonstrated great capabilities of producing
impressive monocular videos, however, the generation of 3D stereoscopic video
remains under-explored. We propose a pose-free and training-free approach for
generating 3D stereoscopic videos using an off-the-shelf monocular video
generation model. Our method warps a generated monocular video into camera
views on stereoscopic baseline using estimated video depth, and employs a novel
frame matrix video inpainting framework. The framework leverages the video
generation model to inpaint frames observed from different timestamps and
views. This effective approach generates consistent and semantically coherent
stereoscopic videos without scene optimization or model fine-tuning. Moreover,
we develop a disocclusion boundary re-injection scheme that further improves
the quality of video inpainting by alleviating the negative effects propagated
from disoccluded areas in the latent space. We validate the efficacy of our
proposed method by conducting experiments on videos from various generative
models, including Sora [4 ], Lumiere [2], WALT [8 ], and Zeroscope [ 42]. The
experiments demonstrate that our method has a significant improvement over
previous methods. The code will be released at
https://daipengwa.github.io/SVG_ProjectPage.

使用现成的单目视频生成模型，我们提出了一种无姿态、无需训练的方法来生成 3D 立体视频，并采用了新颖的帧矩阵视频修复框架。我们的方法利用估计的视频深度将生成的单目视频变形为立体基线上的摄像机视图，并使用视频生成模型对观察到的不同时间戳和视图的帧进行修复。通过减轻潜在空间中从不可遮挡区域传播的负面影响，我们还开发了一种去遮挡边界再注入方案来进一步提高视频修复的质量。我们对包括 Sora、Lumiere、WALT 和 Zeroscope 等各种生成模型的视频进行了实验证明我们提出的方法在之前的方法基础上具有显著的改进。

SVG：通过去噪帧矩阵生成 3D 立体视频

SVG: 3D Stereoscopic Video Generation via Denoising Frame Matrix

Are camera poses necessary for multi-view 3D modeling? Existing approaches
predominantly assume access to accurate camera poses. While this assumption
might hold for dense views, accurately estimating camera poses for sparse views
is often elusive. Our analysis reveals that noisy estimated poses lead to
degraded performance for existing sparse-view 3D modeling methods. To address
this issue, we present LEAP, a novel pose-free approach, therefore challenging
the prevailing notion that camera poses are indispensable. LEAP discards
pose-based operations and learns geometric knowledge from data. LEAP is
equipped with a neural volume, which is shared across scenes and is
parameterized to encode geometry and texture priors. For each incoming scene,
we update the neural volume by aggregating 2D image features in a
feature-similarity-driven manner. The updated neural volume is decoded into the
radiance field, enabling novel view synthesis from any viewpoint. On both
object-centric and scene-level datasets, we show that LEAP significantly
outperforms prior methods when they employ predicted poses from
state-of-the-art pose estimators. Notably, LEAP performs on par with prior
approaches that use ground-truth poses while running $400\times$ faster than
PixelNeRF. We show LEAP generalizes to novel object categories and scenes, and
learns knowledge closely resembles epipolar geometry. Project page:
this https URL

摘要：本研究论文探讨了在多视角三维建模中，相机姿势是否是必需的问题，并提出了一种名为 LEAP 的新型无姿势方法，该方法通过数据学习几何知识并使用神经体积进行图像聚合，展示了在预测姿势之下，LEAP 相较于现有方法有明显的性能优势，同时运行速度比 PixelNeRF 快 400 倍。