Large language models (LLMs) have significantly advanced various natural
language processing tasks, but deploying them remains computationally
expensive. Knowledge distillation (KD) is a promising solution, enabling the
transfer of capabilities from larger teacher LLMs to more compact student
models. Particularly, sequence-level KD, which distills rationale-based
reasoning processes instead of merely final outcomes, shows great potential in
enhancing students' reasoning capabilities. However, current methods struggle
with sequence level KD under long-tailed data distributions, adversely
affecting generalization on sparsely represented domains. We introduce the
Multi-Stage Balanced Distillation (BalDistill) framework, which iteratively
balances training data within a fixed computational budget. By dynamically
selecting representative head domain examples and synthesizing tail domain
examples, BalDistill achieves state-of-the-art performance across diverse
long-tailed datasets, enhancing both the efficiency and efficacy of the
distilled models.

大语言模型（LLMs）在各种自然语言处理任务中有了显著的进展，但部署仍然需要大量的计算资源。我们介绍了一种名为 Multi-Stage Balanced Distillation（BalDistill）的框架，通过在固定的计算资源预算内动态选择代表性的正样本和合成尾部样本，平衡训练数据，并在各种长尾数据集上取得了最先进的性能，提高了蒸馏模型的效率和效果。

多阶段均衡蒸馏：解决序列级知识蒸馏中的长尾挑战

Multi-Stage Balanced Distillation: Addressing Long-Tail Challenges in  Sequence-Level Knowledge Distillation

Learning an effective representation in multi-label text classification
(MLTC) is a significant challenge in NLP. This challenge arises from the
inherent complexity of the task, which is shaped by two key factors: the
intricate connections between labels and the widespread long-tailed
distribution of the data. To overcome this issue, one potential approach
involves integrating supervised contrastive learning with classical supervised
loss functions. Although contrastive learning has shown remarkable performance
in multi-class classification, its impact in the multi-label framework has not
been thoroughly investigated. In this paper, we conduct an in-depth study of
supervised contrastive learning and its influence on representation in MLTC
context. We emphasize the importance of considering long-tailed data
distributions to build a robust representation space, which effectively
addresses two critical challenges associated with contrastive learning that we
identify: the "lack of positives" and the "attraction-repulsion imbalance".
Building on this insight, we introduce a novel contrastive loss function for
MLTC. It attains Micro-F1 scores that either match or surpass those obtained
with other frequently employed loss functions, and demonstrates a significant
improvement in Macro-F1 scores across three multi-label datasets.

我们进行了关于有监督对比学习在多标签文本分类中的影响以及如何构建稳健的表示空间的研究，提出了一种新的对比损失函数，并在三个多标签数据集上实现了显著的宏 F1 分数改进。

探索对比学习用于长尾多标签文本分类

Exploring Contrastive Learning for Long-Tailed Multi-Label Text  Classification

Federated learning is designed to enhance data security and privacy, but
faces challenges when dealing with heterogeneous data in long-tailed and
non-IID distributions. This paper explores an overlooked scenario where tail
classes are sparsely distributed over a few clients, causing the models trained
with these classes to have a lower probability of being selected during client
aggregation, leading to slower convergence rates and poorer model performance.
To address this issue, we propose a two-stage Decoupled Federated learning
framework using Feature Statistics (DFL-FS). In the first stage, the server
estimates the client's class coverage distributions through masked local
feature statistics clustering to select models for aggregation to accelerate
convergence and enhance feature learning without privacy leakage. In the second
stage, DFL-FS employs federated feature regeneration based on global feature
statistics and utilizes resampling and weighted covariance to calibrate the
global classifier to enhance the model's adaptability to long-tailed data
distributions. We conducted experiments on CIFAR10-LT and CIFAR100-LT datasets
with various long-tailed rates. The results demonstrate that our method
outperforms state-of-the-art methods in both accuracy and convergence rate.

本文探讨了尾部类别在几个客户端上稀疏分布的被忽视的情景，提出了一个基于功能统计的两阶段解耦合联邦学习框架（DFL-FS），用于加速收敛并提高特征学习能力，实验结果表明该方法在准确率和收敛速度上优于现有方法。