Batch Normalization (BN) and its variants has been extensively studied for
neural nets in various computer vision tasks, but relatively little work has
been dedicated to studying the effect of BN in continual learning. To that end,
we develop a new update patch for BN, particularly tailored for the
exemplar-based class-incremental learning (CIL). The main issue of BN in CIL is
the imbalance of training data between current and past tasks in a mini-batch,
which makes the empirical mean and variance as well as the learnable affine
transformation parameters of BN heavily biased toward the current task --
contributing to the forgetting of past tasks. While one of the recent BN
variants has been developed for "online" CIL, in which the training is done
with a single epoch, we show that their method does not necessarily bring gains
for "offline" CIL, in which a model is trained with multiple epochs on the
imbalanced training data. The main reason for the ineffectiveness of their
method lies in not fully addressing the data imbalance issue, especially in
computing the gradients for learning the affine transformation parameters of
BN. Accordingly, our new hyperparameter-free variant, dubbed as Task-Balanced
BN (TBBN), is proposed to more correctly resolve the imbalance issue by making
a horizontally-concatenated task-balanced batch using both reshape and repeat
operations during training. Based on our experiments on class incremental
learning of CIFAR-100, ImageNet-100, and five dissimilar task datasets, we
demonstrate that our TBBN, which works exactly the same as the vanilla BN in
the inference time, is easily applicable to most existing exemplar-based
offline CIL algorithms and consistently outperforms other BN variants.

本研究通过针对基于样本的类增量学习（CIL）任务开发了一种新的 Batch Normalization（BN）更新模式 ——Task-Balanced BN（TBBN），通过使用 reshape 和 repeat 操作以及水平拼接任务平衡批来更好地解决训练数据不平衡问题，实验结果表明 TBBN 是一种超参数自由的变体，并且在 CIFAR-100，ImageNet-100 和五个不同的任务数据集上都优于其他的 BN 变体。

基于范例的类增量学习的归一化平衡化

Rebalancing Batch Normalization for Exemplar-based Class-Incremental  Learning

In this paper, we focus on the imbalance issue, which is rarely studied in
aspect term extraction and aspect sentiment classification when regarding them
as sequence labeling tasks. Besides, previous works usually ignore the
interaction between aspect terms when labeling polarities. We propose a
GRadient hArmonized and CascadEd labeling model (GRACE) to solve these
problems. Specifically, a cascaded labeling module is developed to enhance the
interchange between aspect terms and improve the attention of sentiment tokens
when labeling sentiment polarities. The polarities sequence is designed to
depend on the generated aspect terms labels. To alleviate the imbalance issue,
we extend the gradient harmonized mechanism used in object detection to the
aspect-based sentiment analysis by adjusting the weight of each label
dynamically. The proposed GRACE adopts a post-pretraining BERT as its backbone.
Experimental results demonstrate that the proposed model achieves consistency
improvement on multiple benchmark datasets and generates state-of-the-art
results.

本文提出了 GRACE 模型，通过引入层级标签和动态调节标签权重机制，解决了既往研究所忽视的方面术语交互和情感分类不平衡问题。实验结果表明，该模型在多个基准数据集上都取得了最新的、一致的最佳结果。