Classification on long-tailed distributed data is a challenging problem,
which suffers from serious class-imbalance and hence poor performance on tail
classes with only a few samples. Owing to this paucity of samples, learning on
the tail classes is especially challenging for the fine-tuning when
transferring a pretrained model to a downstream task. In this work, we present
a simple modification of standard fine-tuning to cope with these challenges.
Specifically, we propose a two-stage fine-tuning: we first fine-tune the final
layer of the pretrained model with class-balanced reweighting loss, and then we
perform the standard fine-tuning. Our modification has several benefits: (1) it
leverages pretrained representations by only fine-tuning a small portion of the
model parameters while keeping the rest untouched; (2) it allows the model to
learn an initial representation of the specific task; and importantly (3) it
protects the learning of tail classes from being at a disadvantage during the
model updating. We conduct extensive experiments on synthetic datasets of both
two-class and multi-class tasks of text classification as well as a real-world
application to ADME (i.e., absorption, distribution, metabolism, and excretion)
semantic labeling. The experimental results show that the proposed two-stage
fine-tuning outperforms both fine-tuning with conventional loss and fine-tuning
with a reweighting loss on the above datasets.

本研究提出一种两阶段的微调神经网络的方法。首先，利用平衡权重重新设计模型进行初步的调整，然后再进行标准的微调过程，以解决在长尾分布数据上进行神经网络训练时类别不平衡和类别样本不足的问题。实验结果表明，这种方法在各种合成数据集和现实应用中都具有优越性。