LiDAR-based 3D object detection, semantic segmentation, and panoptic
segmentation are usually implemented in specialized networks with distinctive
architectures that are difficult to adapt to each other. This paper presents
LidarMultiNet, a LiDAR-based multi-task network that unifies these three major
LiDAR perception tasks. Among its many benefits, a multi-task network can
reduce the overall cost by sharing weights and computation among multiple
tasks. However, it typically underperforms compared to independently combined
single-task models. The proposed LidarMultiNet aims to bridge the performance
gap between the multi-task network and multiple single-task networks. At the
core of LidarMultiNet is a strong 3D voxel-based encoder-decoder architecture
with a Global Context Pooling (GCP) module extracting global contextual
features from a LiDAR frame. Task-specific heads are added on top of the
network to perform the three LiDAR perception tasks. More tasks can be
implemented simply by adding new task-specific heads while introducing little
additional cost. A second stage is also proposed to refine the first-stage
segmentation and generate accurate panoptic segmentation results. LidarMultiNet
is extensively tested on both Waymo Open Dataset and nuScenes dataset,
demonstrating for the first time that major LiDAR perception tasks can be
unified in a single strong network that is trained end-to-end and achieves
state-of-the-art performance. Notably, LidarMultiNet reaches the official 1st
place in the Waymo Open Dataset 3D semantic segmentation challenge 2022 with
the highest mIoU and the best accuracy for most of the 22 classes on the test
set, using only LiDAR points as input. It also sets the new state-of-the-art
for a single model on the Waymo 3D object detection benchmark and three
nuScenes benchmarks.

本文介绍了一种基于 LiDAR 的多任务网络 LidarMultiNet，它将 LiDAR 的三个主要感知任务：3D 物体检测，语义分割和全景分割统一起来。通过使用全局上下文汇聚（GCP）模块提取全局上下文特征，任务特定的头被添加到网络的顶部执行三种任务。LidarMultiNet 在 Waymo Open Dataset 和 nuScenes 数据集 上进行了广泛的测试，表明主要 LiDAR 感知任务可以在单个强网络中统一，并在既有 API 上取得最佳结果。

LidarMultiNet: 面向 LiDAR 感知的统一多任务网络

LidarMultiNet: Towards a Unified Multi-Task Network for LiDAR Perception

We propose a novel variant of SGD customized for training network
architectures that support anytime behavior: such networks produce a series of
increasingly accurate outputs over time. Efficient architectural designs for
these networks focus on re-using internal state; subnetworks must produce
representations relevant for both immediate prediction as well as refinement by
subsequent network stages. We consider traditional branched networks as well as
a new class of recursively nested networks. Our new optimizer, Orthogonalized
SGD, dynamically re-balances task-specific gradients when training a multitask
network. In the context of anytime architectures, this optimizer projects
gradients from later outputs onto a parameter subspace that does not interfere
with those from earlier outputs. Experiments demonstrate that training with
Orthogonalized SGD significantly improves generalization accuracy of anytime
networks.

提出了一种新的 SGD 算法 -- 正交 SGD，用于训练支持任意行为的网络架构，通过重用内部状态，可以在时间上产生一系列越来越准确的输出，该算法可以在任务特定梯度之间动态地平衡，并显著提高网络的泛化性能。