The evaluation of large language models (LLMs) has drawn substantial
attention in the field recently. This work focuses on evaluating LLMs in a
Chinese context, specifically, for Traditional Chinese which has been largely
underrepresented in existing benchmarks. We present TMLU, a holistic evaluation
suit tailored for assessing the advanced knowledge and reasoning capability in
LLMs, under the context of Taiwanese Mandarin. TMLU consists of an array of 37
subjects across social science, STEM, humanities, Taiwan-specific content, and
others, ranging from middle school to professional levels. In addition, we
curate chain-of-thought-like few-shot explanations for each subject to
facilitate the evaluation of complex reasoning skills. To establish a
comprehensive baseline, we conduct extensive experiments and analysis on 24
advanced LLMs. The results suggest that Chinese open-weight models demonstrate
inferior performance comparing to multilingual proprietary ones, and
open-weight models tailored for Taiwanese Mandarin lag behind the
Simplified-Chinese counterparts. The findings indicate great headrooms for
improvement, and emphasize the goal of TMLU to foster the development of
localized Taiwanese-Mandarin LLMs. We release the benchmark and evaluation
scripts for the community to promote future research.

该研究针对传统汉语在现有基准测试中的低覆盖率，提出了适用于评估大型语言模型的 TMLU 综合评估工具，通过中学到专业水平的 37 个科目以及对每个科目进行链式思考式的少样本解释，基于 24 个优秀的语言模型的广泛实验证明汉语公开权重模型在复杂推理能力上表现较差，而适用于台湾国语的开放权重模型也相对于简体中文版本存在差距，研究发现存在提升潜力，强调了培养本土化台湾国语大型语言模型的目标，并公开了基准测试和评估脚本以促进未来研究。