Bird's-eye view (BEV) maps are an important geometrically structured
representation widely used in robotics, in particular self-driving vehicles and
terrestrial robots. Existing algorithms either require depth information for
the geometric projection, which is not always reliably available, or are
trained end-to-end in a fully supervised way to map visual first-person
observations to BEV representation, and are therefore restricted to the output
modality they have been trained for. In contrast, we propose a new model
capable of performing zero-shot projections of any modality available in a
first person view to the corresponding BEV map. This is achieved by
disentangling the geometric inverse perspective projection from the modality
transformation, eg. RGB to occupancy. The method is general and we showcase
experiments projecting to BEV three different modalities: semantic
segmentation, motion vectors and object bounding boxes detected in first
person. We experimentally show that the model outperforms competing methods, in
particular the widely used baseline resorting to monocular depth estimation.

提出了一种新模型，能够对任何第一人称视角的感知模态进行零样本投影到对应的鸟瞰图，该模型通过将几何逆透视投影与模态转换进行解耦合，实验结果表明该模型优于竞争方法，尤其是单目深度估计。

Zero-BEV：任何第一人称模态的零射击投影到 BEV 地图

Zero-BEV: Zero-shot Projection of Any First-Person Modality to BEV Maps

Talk2BEV is a large vision-language model (LVLM) interface for bird's-eye
view (BEV) maps in autonomous driving contexts. While existing perception
systems for autonomous driving scenarios have largely focused on a pre-defined
(closed) set of object categories and driving scenarios, Talk2BEV blends recent
advances in general-purpose language and vision models with BEV-structured map
representations, eliminating the need for task-specific models. This enables a
single system to cater to a variety of autonomous driving tasks encompassing
visual and spatial reasoning, predicting the intents of traffic actors, and
decision-making based on visual cues. We extensively evaluate Talk2BEV on a
large number of scene understanding tasks that rely on both the ability to
interpret free-form natural language queries, and in grounding these queries to
the visual context embedded into the language-enhanced BEV map. To enable
further research in LVLMs for autonomous driving scenarios, we develop and
release Talk2BEV-Bench, a benchmark encompassing 1000 human-annotated BEV
scenarios, with more than 20,000 questions and ground-truth responses from the
NuScenes dataset.

Talk2BEV 是一个面向自动驾驶环境中鸟瞰地图的大规模视觉语言模型接口。它将语言和视觉模型与 BEV 结构化地图相结合，消除了任务特定模型的需求，从而能够满足各种自动驾驶任务，包括视觉和空间推理、预测交通参与者的意图以及基于视觉线索的决策。在大量场景理解任务上对 Talk2BEV 进行了广泛评估，并发布了包含 1000 个人工注释的 BEV 场景的基准数据集 Talk2BEV-Bench，其中包含超过 20,000 个问题和真实答案。