In this technical report, we present CarLLaVA, a Vision Language Model (VLM)
for autonomous driving, developed for the CARLA Autonomous Driving Challenge
2.0. CarLLaVA uses the vision encoder of the LLaVA VLM and the LLaMA
architecture as backbone, achieving state-of-the-art closed-loop driving
performance with only camera input and without the need for complex or
expensive labels. Additionally, we show preliminary results on predicting
language commentary alongside the driving output. CarLLaVA uses a
semi-disentangled output representation of both path predictions and waypoints,
getting the advantages of the path for better lateral control and the waypoints
for better longitudinal control. We propose an efficient training recipe to
train on large driving datasets without wasting compute on easy, trivial data.
CarLLaVA ranks 1st place in the sensor track of the CARLA Autonomous Driving
Challenge 2.0 outperforming the previous state of the art by 458% and the best
concurrent submission by 32.6%.

CarLLaVA 是一种用于自主驾驶的视觉语言模型，通过 LLaMA 架构和 LLaVA VLM 的视觉编码器实现闭环驾驶性能的最新进展，无需复杂或昂贵的标签，并通过行驶输出展示了预测语言评论的初步结果。