Classification tasks in NLP are typically addressed by selecting a pre-trained language model (PLM) from a model hub, and fine-tuning it for the task at hand. However, given the very large number of PLMs that are currently available, a practical challenge is to determine which of them will perform best for a specific downstream task. With this paper, we introduce TransformerRanker, a lightweight library that efficiently ranks PLMs for classification tasks without the need for computationally costly fine-tuning. Our library implements current approaches for transferability estimation (LogME, H-Score, kNN), in combination with layer aggregation options, which we empirically showed to yield state-of-the-art rankings of PLMs (Garbas et al., 2024). We designed the interface to be lightweight and easy to use, allowing users to directly connect to the HuggingFace Transformers and Dataset libraries. Users need only select a downstream classification task and a list of PLMs to create a ranking of likely best-suited PLMs for their task. We make TransformerRanker available as a pip-installable open-source library https://github.com/flairNLP/transformer-ranker.

本研究解决了在众多预训练语言模型中，如何选择最佳模型以适应特定下游分类任务的难题。提出了TransformerRanker，一个轻量级库，通过高效排名来评估语言模型的适应性，避免了代价高昂的微调过程。实验结果表明，该方法在模型排名上达到了最先进的水平，具有显著的应用价值。

TransformerRanker：有效寻找最适合下游分类任务的语言模型的工具