As Large Language Models (LLMs) are becoming prevalent in various fields,
there is an urgent need for improved NLP benchmarks that encompass all the
necessary knowledge of individual discipline. Many contemporary benchmarks for
foundational models emphasize a broad range of subjects but often fall short in
presenting all the critical subjects and encompassing necessary professional
knowledge of them. This shortfall has led to skewed results, given that LLMs
exhibit varying performance across different subjects and knowledge areas. To
address this issue, we present psybench, the first comprehensive Chinese
evaluation suite that covers all the necessary knowledge required for graduate
entrance exams. psybench offers a deep evaluation of a model's strengths and
weaknesses in psychology through multiple-choice questions. Our findings show
significant differences in performance across different sections of a subject,
highlighting the risk of skewed results when the knowledge in test sets is not
balanced. Notably, only the ChatGPT model reaches an average accuracy above
$70\%$, indicating that there is still plenty of room for improvement. We
expect that psybench will help to conduct thorough evaluations of base models'
strengths and weaknesses and assist in practical application in the field of
psychology.

LLMs 在各领域越来越普及，因此迫切需要改进的 NLP 基准来涵盖各个学科的必要知识。为了解决这个问题，作者提出了 psybench，这是第一个详尽覆盖研究领域所需知识的全面中文评估套件，通过多项选择题深入评估模型在心理学中的优势和劣势，并发现不同领域的性能存在显著差异，展示了测试集中知识不均衡可能导致结果偏斜的风险，仅 ChatGPT 模型的平均准确率超过 70％，表明仍有很大的改进空间。总之，psybench 将有助于对基本模型的优势和劣势进行全面评估，并在心理学领域的实际应用中提供支持。

PsyBench: 一个平衡且深入的心理学中文评估基准

PsyBench: a balanced and in-depth Psychological Chinese Evaluation  Benchmark for Foundation Models

Amid ongoing health crisis, there is a growing necessity to discern possible
signs of Wellness Dimensions (WD) manifested in self-narrated text. As the
distribution of WD on social media data is intrinsically imbalanced, we
experiment the generative NLP models for data augmentation to enable further
improvement in the pre-screening task of classifying WD. To this end, we
propose a simple yet effective data augmentation approach through prompt-based
Generative NLP models, and evaluate the ROUGE scores and syntactic/semantic
similarity among existing interpretations and augmented data. Our approach with
ChatGPT model surpasses all the other methods and achieves improvement over
baselines such as Easy-Data Augmentation and Backtranslation. Introducing data
augmentation to generate more training samples and balanced dataset, results in
the improved F-score and the Matthew's Correlation Coefficient for upto 13.11%
and 15.95%, respectively.

本文提出了一种基于生成式 NLP 模型的简单而有效的数据增强方法，称为 prompt-based，应用于 Wellness Dimensions 的分类预筛选任务，该方法不仅优于 Easy-Data Augmentation 和 Backtranslation 等基线方法，而且在 F 分数和 Matthew 的相关系数方面均实现了明显的改进。