State-of-the-art pre-trained image models predominantly adopt a two-stage
approach: initial unsupervised pre-training on large-scale datasets followed by
task-specific fine-tuning using Cross-Entropy loss~(CE). However, it has been
demonstrated that CE can compromise model generalization and stability. While
recent works employing contrastive learning address some of these limitations
by enhancing the quality of embeddings and producing better decision
boundaries, they often overlook the importance of hard negative mining and rely
on resource intensive and slow training using large sample batches. To counter
these issues, we introduce a novel approach named CLCE, which integrates
Label-Aware Contrastive Learning with CE. Our approach not only maintains the
strengths of both loss functions but also leverages hard negative mining in a
synergistic way to enhance performance. Experimental results demonstrate that
CLCE significantly outperforms CE in Top-1 accuracy across twelve benchmarks,
achieving gains of up to 3.52% in few-shot learning scenarios and 3.41% in
transfer learning settings with the BEiT-3 model. Importantly, our proposed
CLCE approach effectively mitigates the dependency of contrastive learning on
large batch sizes such as 4096 samples per batch, a limitation that has
previously constrained the application of contrastive learning in
budget-limited hardware environments.

我们介绍了一种名为 CLCE 的新方法，将标签感知对比学习与 CE 相结合，通过有效的负样本挖掘增强性能，在少样本学习和迁移学习中使得 CLCE 显著优于 CE。