As autonomous driving technology matures, end-to-end methodologies have
emerged as a leading strategy, promising seamless integration from perception
to control via deep learning. However, existing systems grapple with challenges
such as unexpected open set environments and the complexity of black-box
models. At the same time, the evolution of deep learning introduces larger,
multimodal foundational models, offering multi-modal visual and textual
understanding. In this paper, we harness these multimodal foundation models to
enhance the robustness and adaptability of autonomous driving systems, enabling
out-of-distribution, end-to-end, multimodal, and more explainable autonomy.
Specifically, we present an approach to apply end-to-end open-set (any
environment/scene) autonomous driving that is capable of providing driving
decisions from representations queryable by image and text. To do so, we
introduce a method to extract nuanced spatial (pixel/patch-aligned) features
from transformers to enable the encapsulation of both spatial and semantic
features. Our approach (i) demonstrates unparalleled results in diverse tests
while achieving significantly greater robustness in out-of-distribution
situations, and (ii) allows the incorporation of latent space simulation (via
text) for improved training (data augmentation via text) and policy debugging.
We encourage the reader to check our explainer video at
this https URL&feature=youtu.be and to view the
code and demos on our project webpage at this https URL

通过使用多模态基础模型，我们提出了一种方法来增强自主驾驶系统的鲁棒性和适应性，实现了开放环境中的端到端多模态自主驾驶，并能从图像和文本的表示中提供驾驶决策。