With the increasing popularity of autonomous driving based on the powerful
and unified bird's-eye-view (BEV) representation, a demand for high-quality and
large-scale multi-view video data with accurate annotation is urgently
required. However, such large-scale multi-view data is hard to obtain due to
expensive collection and annotation costs. To alleviate the problem, we propose
a spatial-temporal consistent diffusion framework DrivingDiffusion, to generate
realistic multi-view videos controlled by 3D layout. There are three challenges
when synthesizing multi-view videos given a 3D layout: How to keep 1)
cross-view consistency and 2) cross-frame consistency? 3) How to guarantee the
quality of the generated instances? Our DrivingDiffusion solves the problem by
cascading the multi-view single-frame image generation step, the single-view
video generation step shared by multiple cameras, and post-processing that can
handle long video generation. In the multi-view model, the consistency of
multi-view images is ensured by information exchange between adjacent cameras.
In the temporal model, we mainly query the information that needs attention in
subsequent frame generation from the multi-view images of the first frame. We
also introduce the local prompt to effectively improve the quality of generated
instances. In post-processing, we further enhance the cross-view consistency of
subsequent frames and extend the video length by employing temporal sliding
window algorithm. Without any extra cost, our model can generate large-scale
realistic multi-camera driving videos in complex urban scenes, fueling the
downstream driving tasks. The code will be made publicly available.

通过 DrivingDiffusion 框架，在复杂的城市场景中生成大规模、逼真的多相机自动驾驶视频，无需额外成本。

基于布局导向的多视角驾驶场景视频生成与潜在扩散模型

DrivingDiffusion: Layout-Guided multi-view driving scene video  generation with latent diffusion model

Content creation, central to applications such as virtual reality, can be a
tedious and time-consuming. Recent image synthesis methods simplify this task
by offering tools to generate new views from as little as a single input image,
or by converting a semantic map into a photorealistic image. We propose to push
the envelope further, and introduce Generative View Synthesis (GVS), which can
synthesize multiple photorealistic views of a scene given a single semantic
map. We show that the sequential application of existing techniques, e.g.,
semantics-to-image translation followed by monocular view synthesis, fail at
capturing the scene's structure. In contrast, we solve the semantics-to-image
translation in concert with the estimation of the 3D layout of the scene, thus
producing geometrically consistent novel views that preserve semantic
structures. We first lift the input 2D semantic map onto a 3D layered
representation of the scene in feature space, thereby preserving the semantic
labels of 3D geometric structures. We then project the layered features onto
the target views to generate the final novel-view images. We verify the
strengths of our method and compare it with several advanced baselines on three
different datasets. Our approach also allows for style manipulation and image
editing operations, such as the addition or removal of objects, with simple
manipulations of the input style images and semantic maps respectively. Visit
the project page at this https URL

本文提出一种叫做 GVS 的方法，可基于输入的语义地图，综合出多个真实感的场景视角，还可进行风格操作和图像编辑操作，如通过简单处理输入的风格图像和语义映射来添加或删除对象等。

生成视角合成：从单视角语义到新视角图像

Generative View Synthesis: From Single-view Semantics to Novel-view  Images

Recent approaches for predicting layouts from 360 panoramas produce excellent
results. These approaches build on a common framework consisting of three
steps: a pre-processing step based on edge-based alignment, prediction of
layout elements, and a post-processing step by fitting a 3D layout to the
layout elements. Until now, it has been difficult to compare the methods due to
multiple different design decisions, such as the encoding network (e.g. SegNet
or ResNet), type of elements predicted (e.g. corners, wall/floor boundaries, or
semantic segmentation), or method of fitting the 3D layout. To address this
challenge, we summarize and describe the common framework, the variants, and
the impact of the design decisions. For a complete evaluation, we also propose
extended annotations for the Matterport3D dataset [3], and introduce two
depth-based evaluation metrics.

该论文总结和描述了从 360 全景图中预测布局的常见框架、变体和设计决策所产生的影响，同时提出了 Matterport3D 数据集的扩展注释和两个基于深度的评估指标。

利用单个 360 度图像对曼哈顿房间布局进行重建：现有技术方法的比较研究

Manhattan Room Layout Reconstruction from a Single 360 image: A  Comparative Study of State-of-the-art Methods

We introduce a network that directly predicts the 3D layout of lanes in a
road scene from a single image. This work marks a first attempt to address this
task with on-board sensing without assuming a known constant lane width or
relying on pre-mapped environments. Our network architecture, 3D-LaneNet,
applies two new concepts: intra-network inverse-perspective mapping (IPM) and
anchor-based lane representation. The intra-network IPM projection facilitates
a dual-representation information flow in both regular image-view and top-view.
An anchor-per-column output representation enables our end-to-end approach
which replaces common heuristics such as clustering and outlier rejection,
casting lane estimation as an object detection problem. In addition, our
approach explicitly handles complex situations such as lane merges and splits.
Results are shown on two new 3D lane datasets, a synthetic and a real one. For
comparison with existing methods, we test our approach on the image-only
tuSimple lane detection benchmark, achieving performance competitive with
state-of-the-art.

该研究通过一种新型网络架构 3D-LaneNet 直接从单张图像中预测道路场景中 3D 车道的布局，并展示了在合成和真实数据集上的表现，其对车道合并和拆分等复杂情况的处理也更加明确，较以往方法有竞争力。