Three-dimensional perception from multi-view cameras is a crucial component
in autonomous driving systems, which involves multiple tasks like 3D object
detection and bird's-eye-view (BEV) semantic segmentation. To improve
perception precision, large image encoders, high-resolution images, and
long-term temporal inputs have been adopted in recent 3D perception models,
bringing remarkable performance gains. However, these techniques are often
incompatible in training and inference scenarios due to computational resource
constraints. Besides, modern autonomous driving systems prefer to adopt an
end-to-end framework for multi-task 3D perception, which can simplify the
overall system architecture and reduce the implementation complexity. However,
conflict between tasks often arises when optimizing multiple tasks jointly
within an end-to-end 3D perception model. To alleviate these issues, we present
an end-to-end framework named HENet for multi-task 3D perception in this paper.
Specifically, we propose a hybrid image encoding network, using a large image
encoder for short-term frames and a small image encoder for long-term temporal
frames. Then, we introduce a temporal feature integration module based on the
attention mechanism to fuse the features of different frames extracted by the
two aforementioned hybrid image encoders. Finally, according to the
characteristics of each perception task, we utilize BEV features of different
grid sizes, independent BEV encoders, and task decoders for different tasks.
Experimental results show that HENet achieves state-of-the-art end-to-end
multi-task 3D perception results on the nuScenes benchmark, including 3D object
detection and BEV semantic segmentation. The source code and models will be
released at this https URL

本文提出了一种名为 HENet 的多任务三维感知的端到端框架，通过使用大图像编码器和小图像编码器以及基于注意机制的时间特征集成模块，融合不同框架提取的特征，并根据每个感知任务的特性利用不同网格大小的 BEV 特征、独立的 BEV 编码器和任务解码器，实现了最先进的端到端多任务三维感知。