In this work we present In-Place Activated Batch Normalization (InPlace-ABN)
- a novel approach to drastically reduce the training memory footprint of
modern deep neural networks in a computationally efficient way. Our solution
substitutes the conventionally used succession of BatchNorm + Activation layers
with a single plugin layer, hence avoiding invasive framework surgery while
providing straightforward applicability for existing deep learning frameworks.
We obtain memory savings of up to 50% by dropping intermediate results and by
recovering required information during the backward pass through the inversion
of stored forward results, with only minor increase (0.8-2%) in computation
time. Also, we demonstrate how frequently used checkpointing approaches can be
made computationally as efficient as InPlace-ABN. In our experiments on image
classification, we demonstrate on-par results on ImageNet-1k with
state-of-the-art approaches. On the memory-demanding task of semantic
segmentation, we report results for COCO-Stuff, Cityscapes and Mapillary
Vistas, obtaining new state-of-the-art results on the latter without additional
training data but in a single-scale and -model scenario. Code can be found at
this https URL .

本文提出了 InPlace-ABN 的方法，通过替代传统的 BatchNorm 和 Activation 层，以及在反向传播中恢复所需的信息来减少现代深度神经网络的占用内存，从而实现对已有深度学习框架的方便应用。使用该方法进行图像分类时，我们在 ImageNet-1k 上展示出与其他先进方法相当的结果，并在 COOC-Stuff、Cityscapes 和 Mapillary Vistas 等需要占用大量内存的语义分割任务中取得了最新的、与单尺度、单模型情况下的最佳结果。