Continual learning entails learning a sequence of tasks and balancing their
knowledge appropriately. With limited access to old training samples, much of
the current work in deep neural networks has focused on overcoming catastrophic
forgetting of old tasks in gradient-based optimization. However, the
normalization layers provide an exception, as they are updated interdependently
by the gradient and statistics of currently observed training samples, which
require specialized strategies to mitigate recency bias. In this work, we focus
on the most popular Batch Normalization (BN) and provide an in-depth
theoretical analysis of its sub-optimality in continual learning. Our analysis
demonstrates the dilemma between balance and adaptation of BN statistics for
incremental tasks, which potentially affects training stability and
generalization. Targeting on these particular challenges, we propose Adaptive
Balance of BN (AdaB$^2$N), which incorporates appropriately a Bayesian-based
strategy to adapt task-wise contributions and a modified momentum to balance BN
statistics, corresponding to the training and testing stages. By implementing
BN in a continual learning fashion, our approach achieves significant
performance gains across a wide range of benchmarks, particularly for the
challenging yet realistic online scenarios (e.g., up to 7.68%, 6.86% and 4.26%
on Split CIFAR-10, Split CIFAR-100 and Split Mini-ImageNet, respectively). Our
code is available at this https URL

持续学习在深度神经网络中的应用及其在适应性平衡方面的理论分析。通过提出适应性平衡的批归一化策略 (AdaB2N)，在各种基准测试中显著提高了性能。