The recent development and success of Large Language Models (LLMs)
necessitate an evaluation of their performance across diverse NLP tasks in
different languages. Although several frameworks have been developed and made
publicly available, their customization capabilities for specific tasks and
datasets are often complex for different users. In this study, we introduce the
LLMeBench framework. Initially developed to evaluate Arabic NLP tasks using
OpenAI's GPT and BLOOM models; it can be seamlessly customized for any NLP task
and model, regardless of language. The framework also features zero- and
few-shot learning settings. A new custom dataset can be added in less than 10
minutes, and users can use their own model API keys to evaluate the task at
hand. The developed framework has been already tested on 31 unique NLP tasks
using 53 publicly available datasets within 90 experimental setups, involving
approximately 296K data points. We plan to open-source the framework for the
community (this https URL). A video demonstrating the
framework is available online (this https URL).

近期大型语言模型（LLMs）的发展和成功需要对其在不同语言的各种 NLP 任务中的性能进行评估。本研究介绍了 LLMeBench 框架，该框架最初是为了使用 OpenAI 的 GPT 和 BLOOM 模型评估阿拉伯语 NLP 任务而开发的，但它可以轻松地定制任何 NLP 任务和模型，无论语言如何。该框架还具有零样本学习和少样本学习的设置。用户可以在不到 10 分钟的时间内添加新的自定义数据集，并使用自己的模型 API 密钥评估所需的任务。该框架已经在 31 个独特的 NLP 任务中进行了测试，涉及 53 个公开可用的数据集和大约 296K 个数据点的 90 个实验设置。我们计划将该框架开源给社区。在线上有演示视频供观看。