We present TMMLU+, a comprehensive dataset designed for the Traditional
Chinese massive multitask language understanding dataset. TMMLU+ is a
multiple-choice question-answering dataset with 66 subjects from elementary to
professional level. Compared to its predecessor, TMMLU, TMMLU+ is six times
larger and boasts a more balanced subject distribution. We included benchmark
results in TMMLU+ from closed-source models and 24 open-weight Chinese large
language models of parameters ranging from 1.8B to 72B. Our findings reveal
that Traditional Chinese models still trail behind their Simplified Chinese
counterparts. Additionally, current large language models have yet to
outperform human performance in average scores. We publicly release our dataset
and the corresponding benchmark source code.

我们发布了 TMMLU+，这是一个为传统中文大规模多任务语言理解数据集而设计的全面数据集。TMMLU + 是一个多项选择问答数据集，包括 66 个从初级到专业水平的学科。与其前身 TMMLU 相比，TMMLU + 的规模增加了六倍，学科分布更加平衡。我们在 TMMLU + 中包含了来自闭源模型和 24 个开源的中文大型语言模型的基准结果，这些模型的参数范围从 1.8B 到 72B 不等。我们的研究发现，传统中文模型仍然落后于其简体中文模型。此外，目前的大型语言模型在平均分上仍未超越人类表现。我们公开发布了我们的数据集和相应的基准源代码。