3D dense captioning stands as a cornerstone in achieving a comprehensive
understanding of 3D scenes through natural language. It has recently witnessed
remarkable achievements, particularly in indoor settings. However, the
exploration of 3D dense captioning in outdoor scenes is hindered by two major
challenges: 1) the \textbf{domain gap} between indoor and outdoor scenes, such
as dynamics and sparse visual inputs, makes it difficult to directly adapt
existing indoor methods; 2) the \textbf{lack of data} with comprehensive
box-caption pair annotations specifically tailored for outdoor scenes. To this
end, we introduce the new task of outdoor 3D dense captioning. As input, we
assume a LiDAR point cloud and a set of RGB images captured by the panoramic
camera rig. The expected output is a set of object boxes with captions. To
tackle this task, we propose the TOD3Cap network, which leverages the BEV
representation to generate object box proposals and integrates Relation
Q-Former with LLaMA-Adapter to generate rich captions for these objects. We
also introduce the TOD3Cap dataset, the largest one to our knowledge for 3D
dense captioning in outdoor scenes, which contains 2.3M descriptions of 64.3K
outdoor objects from 850 scenes. Notably, our TOD3Cap network can effectively
localize and caption 3D objects in outdoor scenes, which outperforms baseline
methods by a significant margin (+9.6 CiDEr@0.5IoU). Code, data, and models are
publicly available at this https URL

室外三维密集字幕生成中存在诸多挑战，如室内和室外场景之间的领域差异、数据匮乏等，本文提出了室外三维密集字幕生成的新任务，并引入使用 BEV 表示的 TOD3Cap 网络来生成物体边界框和丰富的字幕，通过 TOD3Cap 数据集进行评估，结果表明我们的网络在室外场景中能够有效地定位和生成字幕，比基线方法有显著提升（+9.6 CiDEr@0.5IoU）。