Multi-modality fusion and multi-task learning are becoming trendy in 3D
autonomous driving scenario, considering robust prediction and computation
budget. However, naively extending the existing framework to the domain of
multi-modality multi-task learning remains ineffective and even poisonous due
to the notorious modality bias and task conflict. Previous works manually
coordinate the learning framework with empirical knowledge, which may lead to
sub-optima. To mitigate the issue, we propose a novel yet simple multi-level
gradient calibration learning framework across tasks and modalities during
optimization. Specifically, the gradients, produced by the task heads and used
to update the shared backbone, will be calibrated at the backbone's last layer
to alleviate the task conflict. Before the calibrated gradients are further
propagated to the modality branches of the backbone, their magnitudes will be
calibrated again to the same level, ensuring the downstream tasks pay balanced
attention to different modalities. Experiments on large-scale benchmark
nuScenes demonstrate the effectiveness of the proposed method, eg, an absolute
14.4% mIoU improvement on map segmentation and 1.4% mAP improvement on 3D
detection, advancing the application of 3D autonomous driving in the domain of
multi-modality fusion and multi-task learning. We also discuss the links
between modalities and tasks.

多模态融合与多任务学习在三维自动驾驶场景中变得流行，然而，将现有框架扩展到多模态多任务学习领域仍然是无效的甚至有害的，由于恶名昭彰的模态偏差和任务冲突。本研究提出了一个新颖而简单的多层梯度校准学习框架，通过任务和模态之间的优化在任务和模态之间进行优化。实验证明了该方法的有效性。