Online continual learning aims to continuously train neural networks from a
continuous data stream with a single pass-through data. As the most effective
approach, the rehearsal-based methods replay part of previous data. Commonly
used predictors in existing methods tend to generate biased dot-product logits
that prefer to the classes of current data, which is known as a bias issue and
a phenomenon of forgetting. Many approaches have been proposed to overcome the
forgetting problem by correcting the bias; however, they still need to be
improved in online fashion. In this paper, we try to address the bias issue by
a more straightforward and more efficient method. By decomposing the
dot-product logits into an angle factor and a norm factor, we empirically find
that the bias problem mainly occurs in the angle factor, which can be used to
learn novel knowledge as cosine logits. On the contrary, the norm factor
abandoned by existing methods helps remember historical knowledge. Based on
this observation, we intuitively propose to leverage the norm factor to balance
the new and old knowledge for addressing the bias. To this end, we develop a
heuristic approach called unbias experience replay (UER). UER learns current
samples only by the angle factor and further replays previous samples by both
the norm and angle factors. Extensive experiments on three datasets show that
UER achieves superior performance over various state-of-the-art methods. The
code is in this https URL

在线连续学习通过对连续数据流进行单次遍历来不断训练神经网络。通过将点积 logit 分解为角因子和模因子，我们发现偏差问题主要出现在角因子中，并利用模因子来平衡新旧知识以解决偏差问题。在三个数据集上的实验证明，解决偏差问题的启发式方法 UER 相对于其他最先进的方法具有更好的性能。

UER: 在线持续学习的启发式偏差解决方法

UER: A Heuristic Bias Addressing Approach for Online Continual Learning

Continual pretraining is a standard way of building a domain-specific
pretrained language model from a general-domain language model. However,
sequential task training may cause catastrophic forgetting, which affects the
model performance in downstream tasks. In this paper, we propose a continual
pretraining method for the BERT-based model, named CBEAF-Adapting (Chinese
Biomedical Enhanced Attention-FFN Adapting). Its main idea is to introduce a
small number of attention heads and hidden units inside each self-attention
layer and feed-forward network. Using the Chinese biomedical domain as a
running example, we trained a domain-specific language model named
CBEAF-RoBERTa. We conduct experiments by applying models to downstream tasks.
The results demonstrate that with only about 3% of model parameters trained,
our method could achieve about 0.5%, 2% average performance gain compared to
the best performing model in baseline and the domain-specific model,
PCL-MedBERT, respectively. We also examine the forgetting problem of different
pretraining methods. Our method alleviates the problem by about 13% compared to
fine-tuning.

本论文介绍了一种连续预训练方法，称为 CBEAF-Adapting，它的主要思想是在每个自注意力层和前馈网络中引入少量注意头和隐藏单元，有效地解决了领域适应中序列任务训练造成的遗忘问题，以中国生物医学领域为例进行了实验，结果表明我们的方法能够显著提高性能，并且减轻了遗忘问题。